A baseline that made the model look 12 times better than it was

Next to the main app park there is a small side project: a lottery statistics app. It does not try to predict which balls come up. It tests one hypothesis: that some numbers are more popular with players than others, so you can predict roughly how many winners each prize tier will have.

A hypothesis like that only means something if you can falsify it. So the dashboard has a tab that compares the model with a naive baseline. If the model can’t beat the dumb guess, the hypothesis has no support, and the dashboard is built to say so.

For a while that comparison was wrong, and wrong in the model’s favour.

What the dashboard showed

Both the model and the baseline are scored by MAE: the average absolute gap between the predicted and actual number of winners. Lower is better.

The old live baseline took every actual winner count the app had collected and averaged them into a single number. It ignored prize tier. In the 7-of-34 game, “4 correct” has around 300,000 winners in a typical draw and “5 correct” around 22,000. Both went into the same average.

That average fits neither tier. It sits far above the 22,000 tier and far below the 300,000 tier, so the baseline missed by a huge amount every time. According to the task log, this pushed the baseline MAE up to about 124,000 and made the model look about 12 times better than it was. The model predicts each tier separately, so it had an easy opponent.

The commit that fixed it puts it like this (translated from Norwegian): the old baseline “pooled answers across tiers (300k and 22k in the same average) and made the model look 12x better”.

The commit also notes a trap on the other side. An average that includes the answer for the draw being scored is in-sample: the baseline gets to see the answer before it guesses. In the commit’s words, an “in-sample average would conversely have flattered the baseline”. Both versions are unfair. One favours the model and the other favours the baseline.

Where it came from

The offline validation step did it right. It split the draws in time order, 80/20, and its baseline was the training-set average per tier. The live dashboard comparison was built in a separate task, and its baseline was the one that pooled the tiers.

The task log is the file Claude Code is told to keep up to date during the build, and it records the dashboard work. The task log entry for the fix calls it a method error. Neither the commit nor the log says in so many words who wrote that line, so we leave it there. The material also doesn’t record how the error was spotted. The log says only that it was “found and corrected” on 2026-07-15.

The fix

The new baseline is prequential: for each prediction it uses only data that existed before that draw. Specifically:

• It is calculated per prize tier, never across tiers.
• For each draw, the baseline is the expanding average of the actual winner counts from earlier draws in the same tier. The model had the same information when it made its prediction, no more and no less.
• Model MAE and baseline MAE are measured on the same rows. A row counts only if the baseline has history to stand on.
• If no row has a baseline yet, the dashboard compares nothing. It says that an honest comparison needs at least two actual results per tier and asks you to come back after the next draw.
• Under the two numbers, the dashboard now shows how many predictions the comparison covers.

The plain-language glossary entry for “baseline”, which appears as a tooltip in the app, was rewritten to match. It used to say “the historical average of the number of winners”. Now it says the average of earlier draws at the same prize tier, the same information the model had.

The honest numbers

Measured on 2026-07-15 with the corrected method, MAE in number of winners:

Game	Model MAE	Baseline MAE	Rows compared
Eurojackpot	19,145	20,492	15
7-of-34 game	696	1,032	2

The model still leads in both games. For Eurojackpot the gap is about 7 percent, not a factor of 12. The 7-of-34 figure rests on two rows, which is too few to conclude anything. In the commit’s own words, the model leads “but soberly”.

What changed is the size of the lead, and the size is what a reader would have taken away. A model that seems 12 times better than a naive guess invites conclusions that a 7 percent lead on 15 rows does not support.

What we take from it

The dashboard displayed both numbers next to each other.

Comparing the old and new baselines also shows that there are two separate ways to get this wrong. Ignoring structure in the data (tiers) made the baseline too weak. Letting it peek at the answer would have made it too strong. The fix had to deal with both at once.

The rule

A baseline must be prequential: it may only use data from before the prediction it is compared with. It is calculated per tier, or per whatever unit the model itself predicts. It is measured on exactly the same rows as the model. And the numbers are reported soberly, with the row count next to them. If there are too few rows for a fair comparison, the dashboard says so instead of showing a number.