What broke
Dials and weights that never took effect
Between 31 July and 7 August in the stock app, four settings looked right in the code but had no effect in practice. They included a holding-period dial that was ignored, a ramp that no simulation used and weights that could never be learned. This is how we found each one and the rule that came out of it.
The short version
What went wrongFour dials or learning mechanisms in the stock app, including inverse_vol, the holding-period dial, matrix weights and the start-up ramp, looked right in code but had no effect in practice.
How we found itFound through pre-deploy audits and the monkey test, which showed positions held one hour despite a 5-day dial, plus engine-level checks with impact.
The rule nowMeasure each dial's effect through the engine with impact, add a sensitivity row, and check that every code path passes the value.
This post documents something Claude got wrong. The claim rests on a commit or a file, not on memory. All of Claude's mistakes
The stock app runs a small fleet of paper-trading simulations. Each one has a risk profile: passive, balanced or aggressive. Each profile is a set of dials. Between 31 July and 7 August 2026 we found four places where a dial or a learning mechanism looked correct in the code but changed nothing in practice.
Two of the commit messages are written in Claude’s own first person. One of them says all six findings in a pre-deploy audit “were in code I wrote the same day”.
inverse_vol: correct maths, no effect
On 31 July Claude built a sizing mode called inverse_vol. The idea is to give volatile stocks less money and stable stocks more, while total exposure stays the same. The owner asked for several verification rounds before a planned deploy. The audit found that the mode was mathematically inert.
The budget it distributed was “position cap × number of names”. The same position cap was also the hard limit per name. That leaves exactly one solution: every name sits at the cap, which is identical to the default equal_cap mode. All three profiles are built so that positions × cap fits inside the investable share. So this would have been true every time. The commit message notes that the variant could never have differed from its control.
The pure function had ten green unit tests, and every one of them was correct. One behaviour test that ran through the engine caught the problem. The same audit found two related problems:
- Not every code path that builds the candidate list attached volatility, including the one used by the
impactverification tool. - The volatility query had never worked. It threw an error, and a silent catch turned that into an empty result. An empty result is a legitimate state, so nothing failed.
The holding-period dial: five days that held for one hour
Later the same evening, the owner asked for an acid test before deploying. The check we expected to matter came back clean. The real finding came from somewhere else.
The monkey test compares the app’s picks against random ones. It showed positions being bought and sold on the same day, marked as rotation exits. The commit records this:
| Profile | Dial says | Actual hold |
|---|---|---|
| Passive | 5 days | 1.0 hour |
| Balanced (two sims) | 3 days | 1.0 hour |
| Aggressive | 0 days | 1.0 hour (correct) |
The dial existed in all three profiles but never reached the target-portfolio engine, which every simulation runs on. So passive and balanced behaved exactly like aggressive on the one dimension the dial exists to separate. They also paid trading costs twice on decisions that lasted an hour. The commit notes that friction had been measured at 71 % of the underperformance. There was a second-order effect too: the monkey test cannot evaluate same-day trades. The app was making trades that could never be scored for stock-picking skill.
The fix put the dial into the engine’s config and gave each held position a holding time. The rule is enforced in the swap loop. Risk exits such as stop-loss are deliberately not affected (“a loss limit that can be postponed is no loss limit”). An unknown holding time counts as old enough, so missing data cannot lock the portfolio. Holding time is passed from every code path that builds the inputs. Otherwise impact would plan swaps that the engine refuses.
The commit calls this the third time the same class of bug had bitten. It added a table to the docs covering all 31 config fields: 12 reach the engine through the main mapping, 14 through other routes, and 5 are inert. Each inert one comes with a reason: take-profit is deliberately overridden, four P&L-based rotation dials are inert by design, and one is legacy-only.
Matrix weights that could never be learned
On 5 August, the owner asked (translated): “have we now built it so that as soon as signals mature, they are taken into the calculation?”
They were not. The weighting function looks up each signal source’s own outcomes and gives it a measured weight once it has at least 8 matured rows. But only the aggregate was being recorded. The lookup would never find a row for any single source, so every weight would have stayed at 1.0 forever. The commit message, in Claude’s voice (translated): “The mechanism I described as automatic could never kick in. It looked right in the code and was dead in practice.”
The fix records each scorer as its own measurable source. 120 outcomes were backfilled for 31 July to 5 August. A new comparison variant was checked before launch to confirm it could actually differ from its control: its top 10 shared 1 name with the control’s.
The start-up ramp: set in zero simulations
On 7 August the owner asked for a forward-looking sweep, “so we don’t assume anything only to find out we must wait another period” (translated). All 19 invariants passed. Two dials did not.
The start-up ramp limits how many new positions an account opens per day. It had been built that day, unit-tested and checked through the engine with impact, and then it was set in zero simulations. The commit says a sector cap had sat in the same state for six weeks. An A/B test was not possible: the ramp only binds while an account is building up. So it was added to the backtest engine and measured across four historical drawdown episodes, together with a sensitivity row.
The sensitivity row is there so that “looks like the control” cannot be mistaken for dead code, which was the inverse_vol trap. This time the mechanism was alive, and it hurt on both sides. The ramp stays off, and that is now a measured decision rather than a postponed one. The commit makes one more point: impact said “not dead”, and the stress test said “not good”. The first answer alone reads like an approval.
The same sweep found that a signal-based exit rule had never fired in any simulation. It needs a position to survive a 5-day window. The aggressive variant it ran on holds positions for 3.4 days on average, and only 2 of 11 closed positions reached the window.
The rule
Unit tests prove that a function is correct. They do not prove that the engine ever calls it with inputs that change anything. Before a dial or a learning mechanism is switched on:
- Measure its effect through the engine with
impact. Write one behaviour test per dial: the plan with the dial set must differ from the plan without it. - Add a sensitivity row whenever a variant may end up close to its control, so that “looks like the control” is never confused with dead code.
- Check that every code path passes the value. Several parts of the app build the same inputs, including the verification tool. If one of them leaves a field out, the dial is dead there, and in the worst case it is dead inside the tool meant to verify it.