Measurements that answered the wrong question

Between 31 July and 5 August 2026 we found four measurements in the stock app that gave clean numbers and passed their tests, but answered a different question from the one we were asking. None of them crashed. Each one produced a number that looked like a fact about the market when it was really a fact about how we measured.

The commit messages were written during the coding sessions. They quote Henning and describe the misses in the first person, for example: “I gated on days and span, but not on how many different things we have seen.” This post says “we” for the measurement code and quotes the commits where they speak for themselves.

1. Rows instead of decisions

On 2 August 2026 the first exit outcomes matured: the app could now see what a sold stock did in the 30 days after the sale. The raw numbers matched an earlier dry run exactly. The aggregate built from them did not.

The table of exit outcomes has one row per simulation. If several simulations held the same stock and sold it on the same day, that one sale showed up several times. Both the aggregate and the go-live gate counted rows, so each decision counted as many times as there happened to be simulations holding that position. The commit gives the distribution: 57 decisions with one copy, 16 with two, 15 with three, 3 with four and 1 with six.

The copies carried no new information. Among the 35 duplicated decisions, none had a different 30-day return. The commit calls this pseudo-replication. It made the confidence interval look narrower than it was: the first verdict on stop-loss exits read n=3 with an interval of 0.44 to 1.0, for what was really one decision.

Even at that tiny sample the share of correct exits moved: 85.7 % counted in rows, 66.7 % counted in decisions. After the fix, 152 rows became 94 decisions. The count is 94 and not 92 because the replacement stock is part of the key: “sold A, bought X” and “sold A, bought Y” are different decisions, and 2 of the 35 duplicates were exactly that.

The go-live gate got stricter: its required number of matured exits is now counted in decisions rather than rows. The threshold itself did not change, and the commit notes that Henning approved the change. The gate also had a second problem: it judged every exit against the universe median, while the numbers shown next to it judged a swap against the replacement stock. Both now use the same definition.

2. Exposure instead of skill

On 31 July the hypothesis gate was rejecting candidate strategies with the reason “does not beat benchmark”. Henning’s comment, as quoted in the commit (our translation): “important that we measure all sources the same way.”

The gate compared a selective, partly invested rule against a fully invested Norwegian index (OSEBX). Over the test window, 10 October 2025 to 30 July 2026, the index rose well above the median stock in our own universe of 464 instruments. Two hypotheses had been rejected, though they were roughly level with the median stock.

A rule that sits in cash part of the time and has a stop-loss and a maximum holding period will lose to a fully invested index in a strongly rising market, however well it picks stocks. In the commit’s words, the gate measured “exposure, not skill”. It was also the only place in the app that used the index. Every other signal source was already judged against the universe median.

The required excess return was left untouched. Only the benchmark changed, and the commit points out that this can cut both ways: in a period when the median stock beats the index, the median becomes the stricter gate.

3. Elements instead of time

On 5 August Henning asked whether the old-versus-new distinction we had discussed was actually being taken into account. It was not, and the commit says it was worse than unused: the freshness factor sat between 0.97 and 1.00 for every source on the two stocks checked.

The concentration measure was computed over elements, not over time. One stock had five news items, all from the same cluster four days earlier. Five equal weights gave a concentration of 0.00, “evenly spread”, and so freshness 1.00. But all of that information arrived at a single moment, which is as concentrated in time as it gets.

For price windows the two questions give the same answer, because each element is one day. That is why the error never showed there. It only appeared on event sources, which is where age matters most. After grouping by age first, concentration for that stock went from 0.00 to 1.00 and freshness from 1.00 to 0.57. Two new tests now encode cases where we know the answer.

The commit is open about how much this changed: “the fix changes the number, not any decision.” The number of qualifying candidates stayed at 290.

4. A few companies read as broad evidence

On 4 August the evidence gate checked how many days a signal had been observed and the span between first and last observation, but not how many companies were behind it. One signal source was being weighted up on 53 outcomes from 4 days and 14 companies, each counted about 3.8 times. As the commit puts it, 90 days of span on five companies “is not a claim about a strategy — it is a claim about five companies.” The gate now requires at least 30 distinct companies to weight a source up and 50 to bench one.

The next day the same pattern turned up in the milestone checks. A check marked a point “ready” after a single matured outcome. The insight layer showed +6.82 percentage points of excess return over 8 matured outcomes, which looked convincing. But those 8 came from four companies, four of them from a single company, and without that company the result was −0.29 points. A check now needs 20 matured outcomes across 10 instruments, and it reports both numbers.

How they were found

One came to light when real data showed up for the first time (the first matured exits). One came from a comment Henning made and one from a question Henning asked. The breadth problem in the evidence gate came from a deliberate sweep after Henning’s instruction to use the error class “to find the other errors. Go over it once more. EVERYTHING!” The milestone problem turned up the next day, after Henning asked whether the app learns from insights. The sweep produced a name for the class, written into the project’s error-class notes: a number produced by the measuring apparatus is read as if it were produced by the world.

The exit fix was checked against a hand calculation on real data: rotation n=2 at 50 %, stop-loss n=1 at 100 %. A new invariant compares the stored sample size with the number of distinct decisions. To show that it actually catches the problem, the stored value was set back to the row count, and the check failed, reporting a stored sample of 3 against 1 actual decision.

The rule

• Group by decision and count distinct instruments. A row count is not an amount of evidence. Before trusting a number, ask how many independent things are behind it: rows, days and names.
• Use the same yardstick for every source. That yardstick is the universe median. A gate that uses a different benchmark from the numbers shown next to it is answering a different question.
• Verify a measure against a case where you know the answer. That means a hand calculation, a cluster that must score 1, or an invariant you have watched fail. Green tests only show that the code does what it was written to do, not that it measures the right thing.