The self-audit kept telling us to fix Paris when the whole exchange was just down

The stock app runs a self-audit. It looks at how signals have done against the market and turns what it finds into improvement proposals. One of its checks asks whether a whole exchange is underperforming. On 31 July, 6 September, 13 September and 11 October it gave the same answer: Euronext Paris.

The flag behind these proposals had fixed wording. It said Paris was not beating the universe median, that it was being downweighted automatically in the ranking, and that we should check whether entry or exit quality was failing there.

We rejected the fourth proposal on 11 October. The same day, Claude fixed the check that produced it. Three things in that proposal were wrong, and the flag’s own text had supplied two of them.

What the check measured

The check scores an exchange by direction-adjusted hit rate over 30 days. A bullish signal counts as a win if the stock beats the benchmark. A bearish signal counts as a win if the stock does worse than the benchmark. The benchmark is the median of the universe. An exchange gets flagged when, across its matured outcomes, even the upper bound of the Wilson interval on the hit rate is below 50 %.

That sounds reasonable until the exchange as a whole has a bad stretch. If every stock in Paris drifts weaker than the universe, every bullish signal there loses and every bearish signal wins, however good or bad the signals are. In Paris, most signals were bullish (1,680 against 535 bearish), so the total hit rate drops and the flag fires.

This is what the numbers showed when we measured on 11 October, direction-adjusted against the universe median:

Exchange	Bullish	Bearish
Euronext Paris	−1.21 pp (n = 1,680)	+1.80 pp (n = 535)
NASDAQ	+3.18 pp	−2.87 pp
Oslo Børs	+3.49 pp	−2.88 pp

In Paris, bullish signals lost and bearish signals won. NASDAQ and Oslo show the mirror image: bullish won and bearish lost, because those markets moved stronger than the median. Measured as raw excess return against the average, Paris was −2.4 pp for bullish signals and −2.9 pp for bearish ones. That is about the same move whichever way the signal pointed. When the sign flips with the signal direction, you are looking at exchange beta, not at signal quality.

No change to entries, exits or slippage can fix a market that simply went down relative to other markets.

The two premises the flag planted

The other two errors were in the flag’s wording, which Claude wrote.

First, the text said the exchange “is downweighted automatically in the ranking”. It was not. The dimension weight that would do this is off in all simulations. So every proposal started from a false claim that a correction was already in place.

Second, the text asked us to consider whether entry or exit quality was the problem. The outcome data the check reads has no fills and no exits. It compares the signal against the benchmark, nothing more. Exit and slippage cannot explain numbers that never include an exit or a fill.

Our progress log names the flag itself as the root cause. Its text planted the premises, and the proposals built on them. Four times, the audit read its own wording back as a finding.

The fix

The check now looks at bullish and bearish outcomes separately for each exchange, using the same 30-day window and the same benchmark as before. If bullish signals are below zero and bearish signals are above zero, and there are at least 30 outcomes in each direction, it reads that as beta and stays silent.

The flag still fires when signals miss in both directions. That is the case that actually points at the signals. It also still fires when the bearish sample is too thin to tell beta from failure. One of the three new tests covers this: the Paris pattern with only 10 bearish outcomes still raises the flag.

The text changed too. When the flag fires, it now shows the bullish and bearish split with sample sizes. It says the dimension weight is off by default, so the exchange is not downweighted. It says the outcomes are signal against benchmark without fills or exits, and that the first step is to check the rule mix per source. That last point ties back to an earlier finding: an exchange’s numbers can also reflect which rules happen to fire there. Exchange beta is the second confounder of the same kind.

We ran the new check against production the same day. Paris came out as beta, and no exchange was flagged.

The commit that changed the code also added a line to the user-facing changelog. In plain words, it says the self-audit no longer suggests fixing an exchange just because the whole exchange did worse than the rest of the market, that this happened four times with Paris, and that the proposal now only comes when signals miss both ways.

What it cost

The flag becomes an improvement proposal. The code comment Claude wrote with the fix says both false premises went straight into four rejected Paris proposals.

The rule

A performance flag on a group (an exchange, a sector) has to separate the group’s own movement from the signals’ quality before it proposes anything. If the result flips sign with signal direction, it is the market, and the flag stays quiet. A flag’s text may only state what the system actually does. If the text claims a correction is already active, or points to data the measurement doesn’t contain, the proposals it produces will repeat those errors.