Against the median, 52.5% of random sources looked like a proven edge

On 23 September 2026 Henning told Claude that we kept finding errors in the stock app’s learning. He asked for the learning to be done properly this time and gave Claude a free hand. That evening’s work produced one number that changed how we read every earlier result: 52.5%.

Why the errors kept coming back

The stock app learns which signal sources are worth following. Claude went through the whole codebase twice, only reading, and found that statistics about a signal source were calculated in at least 12 places, each with its own definition:

• hit rate was calculated seven different ways
• source weight four ways
• “proven source” four ways
• two jobs wrote different hit rates into the same field

Decisions were also made on rows rather than independent observations. The same stock flagged by the same source day after day counted again each time. One source had 903 rows, which came down to 58 episodes.

Each fix landed in one copy and the other copies drifted on. That is why the same kind of bug kept turning up. The repo’s register of error classes records this as its own class: the same quantity calculated in many copies.

One definition

Claude replaced the copies with a single module, the source verdict. Here is how it works:

• Episodes, not rows. An episode is one source and one stock. A new signal within 30 days of the episode’s start counts as the same observation.
• Hit rate is measured against the universe median in the same window. Its null is 50%.
• Excess return is measured against the universe mean in the same window. Its null is 0.
• Interval: a 95% cluster bootstrap of mean excess return, with the stock as the cluster.
• Leave-one-out: the mean recalculated without the stock that contributed the most.
• Proven edge needs at least 30 episodes, at least 15 stocks, a lower bound above zero and a positive leave-one-out mean. Proven negative needs at least 30 episodes, an upper bound below zero and a negative leave-one-out mean. Anything else is “unknown”. The doc says it plainly: not proven bad is not the same as good.

The status follows the money (mean excess return), not the hit rate.

Testing the definition on known answers

The verdict got 14 tests on synthetic data, where we know the right answer. They include a planted edge, a planted negative, one lucky stock every day for a year, a narrow source, and skewed gains with a low hit rate. Two of the tests are calibration tests, each with 200 noise sources that by construction have no edge. Measured on 23 September:

200 noise sources	false edge	false negative
normal distribution	1%	2.5%
heavy tails (t, df 3)	3.5%	2.5%
same data, old row counting	19%	n/a

The tests also cover the other direction. A real +1 percentage point edge (sd 5%, 80 episodes) is detected 47% of the time. Small edges need more data, and the verdict says “unknown” rather than guess.

The placebo on real data

Synthetic data only proves the arithmetic. To test the whole measuring setup, Claude built a placebo: 200 fake sources that pick random active stocks on the real signal days, using the real outcomes. By construction they have no edge. With a 95% interval on each side, roughly 2.5% false findings in each direction is the expected result.

The placebo can run against either benchmark. Here are the results from 23 September:

Benchmark	false edge	false negative	mean excess return
universe median (what the app used until 23.09)	52.5%	0%	+0.97 pp
universe mean (now)	0.5%	2.0%	+0.01 pp

Against the median, more than half of the random sources came out as a proven edge.

The reason is the shape of the data. Stock returns are right-skewed, so the mean sits above the median, and a random basket beats the median by about one percentage point for free. The median is the right null for hit rate (did the stock beat the typical stock?). The mean is the right null for excess return (did the stock earn more than the average stock?). The app had been measuring excess return against the median as well.

That code was Claude’s, and the register of error classes records it as a fault.

Switching the benchmark moved false edges from 52.5% to 0.5%. The row counting the episodes replaced had its own problem: 19% false edges on synthetic noise.

What changed in the data

A migration added mean benchmarks next to the median ones in the outcomes table. It was backfilled the same evening: 22,106 rows in one pass and 41,838 in another. The hypothesis verdict, live performance, ranking replay and the variant comparison all switched to the mean. The job that had written absolute hit rates into the shared field no longer writes over a signal source.

With the new definition, three of the four sources that were benched on 23 September were not proven bad.

The verdict did not get to make decisions

Before running any replay, we set the bar for letting the new verdict drive weights and benching. Because this corrects a measured error rather than adding a strategy, the requirement was non-inferiority: a portfolio replay could fall no more than 0.5 pp below the current definition in each of two windows (12.08–01.09 and 02.09–22.09).

It failed. In the second window, both settings tested (the balanced base, and the balanced base with the signal matrix) came out more than 0.5 pp below the current definition in the replay, so decisions stay on the old definition. The source verdict is now the measurement truth: it is stored, archived daily and shown on the learning page, but it does not drive trades. Both settings are re-judged every Sunday against the same requirement. The docs give two readings of the result: in the first window almost no outcomes had matured at the time, so nearly every source was “unknown”; and in the matrix, the verdict in practice makes the source weights equal, while the current weights differ.

The time-travel check

The second guard asks whether the app can reproduce its own past. It reruns a logged day’s ranking as of the moment it was logged and demands exactly what was logged.

On the first run on 23 September, the max ranking path matched. The matrix path did not: it had about 300 rank swaps per day, and 1 to 3 stocks moved in or out at the score threshold. The cause was a single “matured at” timestamp shared by all horizons, which moves when the 90-day value fills in, plus a cutoff that was one day too strict. The fix stores a timestamp for each horizon’s value. After that, all 16 of 16 day-and-mode runs over the last 8 days matched exactly, and the matrix path matched for all 40 days.

The guards

Both checks now run in the app’s audit, which went to 27 of 27 checks:

• Placebo: 100 random sources on real data. No more than 6% may be proven edge, no more than 6% proven negative, and the mean excess return must stay within 0.5 pp of zero. This guard exists to catch the benchmark or the outcome data drifting back.
• Time travel: the last 3 days of logged rankings must reproduce exactly in both modes.

The lesson we took: measure the measuring setup, not just the sources. If random sources pass your test for “proven”, the test is broken.