The gate was never code, and pointed the wrong way

The monitoring app runs an object detector on camera streams. Over time it collects its own training set, and the plan was to fine-tune a local model on it to replace the stock COCO base model. One earlier attempt had already lost against the base model. So we wrote a decision, numbered D-26 in the project’s decision log, to keep that from happening quietly again: models change only after an A/B test against our own failure images, never on validation numbers.

The commit of 21 August 2026 describes what actually existed behind that decision. Not much.

What we found

The commit that fixed it opens with the problem in its subject line: “the gate D-26 decided on was never written down as code”.

The body goes on: “D-26 said models change on an A/B against our own failure images, never on validation numbers. Nothing implemented that.” The only trace of the last measurement was a 60-line answer-key file. Nothing in the repository referred to it, and the one-off script that wrote it was gone. In the commit’s words: “A gate you cannot run again is not a gate.”

Henning writes no code himself, so the missing gate, the vanished one-off script and the training script were Claude’s.

The file had three more problems.

It pointed the wrong way. 58 of its 60 samples had been boxed by the base detector itself. An exam built from what the old model found only asks whether the new model agrees with the old one. The decision log puts it more bluntly: the answer is given in advance.

It wasn’t held out of training. Nothing kept it out of training. A candidate model would have been graded on pictures it had trained on. The commit: “the candidate would have been measured on pictures it had memorised.”

It would have been eaten. A nightly cleanup job deletes the oldest samples per camera and verdict. An exam stored in the training folder would have disappeared a piece at a time.

The training recipe had drifted too. The decision named a specific fine-tuning setup: freeze the first ten layers and use a low learning rate. The script didn’t do either. It still trained every layer at the default learning rate, which the commit calls “the exact run that lost last time.”

The earlier measurement also ran the wrong machinery. It used the training library’s own predict function, which letterboxes the image, keeps the aspect ratio and applies its own threshold and non-maximum suppression. The detector in production does none of that. It stretches a square crop to 320×320, divides by 255 and runs its own post-processing. So the old number described a different machine from the one actually running. A comment in the new measurement script mentions a case from D-26 where a model scored 0.857 mAP on validation and found 0 of 64 real images.

What we changed

The exam is frozen once, outside the training folder, where the cleanup job never looks. It has three parts, selected deterministically by hash per camera:

• Regression images: samples today’s detector boxed itself. A new model must not lose these.
• Failure images: cases taken from the app’s own labelling step, where a new model has to do better than the current one. These are the “own failure images” D-26 meant. A new model has to win here.
• Background images with nothing to find. Every detection here is a false alarm.

The training script refuses to run until the exam exists, and it drops every image whose hash appears in the exam. The recipe from the decision is now in the script, and it can still be overridden from outside without editing the file.

The measurement runs inside the recorder’s own container image, calling the recorder’s own region, post-processing and label-loading functions and the same OpenVINO call on the same CPU detector. If the recorder is upgraded, the measurement is upgraded with it. The script exits with a failure code if the candidate doesn’t pass three checks: it keeps the regression hits on crops, it finds more failure images on whole frames, and it raises no more false alarms than the base model at either threshold or in either view.

It reports two views. “Whole frame” is the full image stretched to 320×320. “Crop” is the region the recorder would cut around the object. It also reports two score thresholds: 0.4, the detector’s own cutoff, and 0.7, which a detection must clear to become an object in the recorder.

What the numbers said

The decision log also points out a limit of this exam. Both positive parts are defined by what today’s detector managed, so the base model wins the regression column and loses the failure column by construction. The useful comparisons are a candidate against the base model in the same column, plus the false-alarm columns, which neither part was built from.

As a check on the new setup, we re-ran the candidate model from 19 August. It lost the same way D-26 had described: 35/95 on crops against the base model’s 74/95, and 17 false alarms on whole frames against 7. That gave us some reason to trust the next result.

It still wasn’t finished

The frozen exam fixed what we could see on 21 August, but not everything. It measured full-resolution images, while the detector in production sees a lower-resolution stream. That was corrected in a later decision, D-33.

The rule

A decision isn’t a gate until code enforces it. For model swaps that means three things:

1. Freeze the exam once, outside anything that trains on it or cleans it up, and make the training script refuse to run without it.
2. Build part of the exam from what the current model misses, not only from what it finds. Otherwise the test only checks agreement with the old model.
3. Measure with the same code path the production detector uses, not with the training library’s own predict function.