The exit measurement counted simulations, not decisions

The stock app runs several simulations side by side. Each one holds positions, sells them and sometimes replaces them with something else. To find out whether those sales were good, the app records every exit and comes back 30 days later to see how things turned out. It asks two questions. Did the stock we sold do worse than the stock we bought instead? For a plain sale to cash, did it do worse than the median? Those results feed two things: a per-exit-type accuracy figure with a Wilson confidence interval, and one of the checks in the go-live gate, “≥50 matured exits with a positive hit rate”.

Both of them counted the wrong thing.

What was wrong

The exit outcomes are stored as one row per simulation. When several simulations held the same stock and sold it on the same day, the same sale was in the table several times. The aggregate counted rows, and so did the go-live gate.

The commit that fixed it puts it plainly: “the exit outcomes table has ONE ROW PER SIMULATION. The same sale was therefore in there several times, and both the aggregate and the go-live gate counted rows.” So the weight each decision got depended on how many simulations happened to own the position. The commit lists the spread:

• 57 decisions with 1 copy
• 16 with 2
• 15 with 3
• 3 with 4
• 1 with 6

The copies told us nothing new. Of the 35 duplicated decisions, none had a different 30-day return for the stock that was sold. The commit calls it pseudo-replication: the same observation counted more than once, which makes the confidence interval look narrower than it is.

The first verdict shows how much this mattered. The stop-loss exit type came out as n=3, with an interval of 0.44 to 1.0. In reality that was one decision, so n=1, and the commit message gives the interval as 0.21 to 1.0. (The code comment in the same commit says 0.02 to 1.0. For one success out of one, the standard 95 % Wilson interval starts at about 0.21, so the comment looks like a typo.)

The accuracy figure moved before there was much data at all. The commit message says: “85.7 % correct counted in rows, 66.7 % counted in decisions.”

How it was found

The first exit outcomes matured on 2 August 2026 at 05:00. According to the commit, the numbers matched the earlier dry run exactly, “but the aggregate read them wrong.”

The fix was checked against a hand count, not just against the test suite. The commit is in capitals here: “VERIFIED AGAINST REAL DATA, not just green tests.” After the change, the aggregate gave rotation n=2 at 50 % and stop-loss n=1 at 100 %, which matched the hand count.

The fix

The aggregate and the gate now reduce the rows to one per decision before they count anything. A decision is identified by the instrument, the exit date, the exit type and the replacement instrument.

The replacement has to be part of that key. “Sold A, bought X” and “sold A, bought Y” are different decisions, and 2 of the 35 duplicated cases were exactly that. With the replacement in the key, 152 rows became 94 decisions. Without it, the result would have been 92.

The go-live gate got the same correction, and it made the gate stricter. Before, “≥50 matured exits” would have been met at around 30 real decisions. Now it takes 50. The threshold itself did not change. The commit notes that Henning, as owner, approved the change.

While in that code, Claude found a second mismatch. The gate judged every exit against the median of the stock universe, while the aggregate shown right next to it judges a swap against the stock bought instead. The commit points out that rotation is the most common exit type, “so it was not a detail.” Both now use the same definition.

Two more changes keep the problem visible:

• The stored statistics now include the row count next to the decision count. If the row count is much higher, several simulations held the same position, and nobody has to mistake the row count for the amount of evidence.
• The app’s audit command got a new check that compares the stored sample size with the number of distinct decisions. Claude then confirmed the check works by setting the sample size back to the row count. The audit failed with “stop_loss: stored 3 vs 1” and a non-zero exit code. After restoring it, 12 of 12 checks passed. The commit reports 682 tests green and a clean type check.

It happened again three days later

On 5 August, the first day with open markets after a new deploy, a report on blocked rotations produced its first real data. It tracks swaps the engine wanted to make but did not, and why. It showed 44 cases blocked by the cost threshold and 26 because the exchange was closed.

The report had the same bug. Three identical lines with n=13 looked like duplicates. In fact they were three different simulations looking at the same swap. The commit in Claude’s own words: “THE REPORT REPEATED A MISTAKE I HAVE DOCUMENTED MYSELF.” Nine simulations look at the same swap every cycle, so one decision can produce up to nine rows per run.

Grouped by decision, the 70 “cases” turned out to be four decisions. The biggest was 39 rows from three simulations over seven hours. The report now shows the number of simulations and the number of distinct hours next to the row count, with a note that a swap seen by nine simulations over three hours gives 27 rows but is still one decision.

The second fix also produced a real finding. The engine had wanted to make the same swap for seven hours, and the cost threshold had refused every time. Once both stocks have matured, we can check whether the threshold was right.

The lesson: the report repeated a mistake Claude had already documented after the first incident.

The rule

When several simulations share a table, a row is not a decision. Deduplicate on the decision before counting, with the replacement instrument in the key. Show the row count next to the decision count. Check aggregates against a hand count from the raw data, and add an audit check that fails if the two drift apart.