Tiling worked in a test and never ran in production

The monitoring app has a step that looks through a recorded clip for the best result it can find. On 17 August 2026 we found that the most useful part of that step had never run in production. It had worked when called directly. Under the real job it was skipped every time.

This post covers what went wrong, what the fix changed and the rule we took from it.

What tiling is for

The clip analyser picks frames from a clip, runs a detector on them and keeps the best result. A small target in a large frame barely registers when the detector looks at the whole frame at once.

Tiling deals with this. The analyser cuts the frame into overlapping pieces and runs the detector on each piece, so a small target takes up more of what the detector sees. The analyser only tiles frames whose shorter side is at least 720 pixels.

The job that calls the analyser can also send a box: a rectangle from the recorder that marks where it detected something. The analyser crops to that box and looks inside it first.

What went wrong

Tiling depended on there being no box. The commit message that fixed it says so directly: the tiling worked in a direct test and did nothing in production, because it was gated on the box being absent and the job always sends a box.

A direct call without a box took the tiling path, found results and looked fine. The production job always sends a box, so it never reached that path. The test exercised a path production never uses.

The commit also says the condition was wrong in principle, not just in practice. In its words, the box “describes one instant; a clip is the whole event”.

The box describes one moment. The clip runs for the whole event. Searching only inside the box means searching a spot that may no longer hold what the box marked.

The difference on real clips

The commit records what the fix changed over that morning’s clips: “the difference between nothing and two”, with quality scores of 0.70 and 0.55. A code comment added in the same commit records that the first version found nothing in a clip where the tiles found a result at 0.70.

A second bug underneath

Changing the condition exposed a coordinate bug. In the old code the tiles were cut from the cropped box, and the detection’s position was moved back into the full frame by adding both the tile’s offset and the crop’s offset. In the new code the tiles are cut from the full frame, so each tile has its own corner in full-frame coordinates. Only the box crop sits inside the frame at the crop offset. The fix gives each candidate its own offset: the crop’s offset for the crop, and the tile’s own corner for each tile. Without it, detections found in tiles would have been reported in the wrong place.

What changed

The commit changed the analysis step:

• Tiles now run on every qualifying frame, whether or not a box was sent. The box crop still runs first, and the best result from either one wins.
• Tiles are cut from the whole frame, not from inside the box.
• The default number of frames sampled from a clip went from 12 to 20.

The same commit fixed an unrelated gap in the worker. When the recorder had no footage yet for an event, it could answer with an empty body and a success status. The clip fetcher accepted that, stored an empty file as a recording and passed the analyser a file OpenCV cannot open. The snapshot fetcher already checked for empty bodies and the clip fetcher did not. Now the clip fetcher treats an empty body as “recorder unavailable” and stores nothing.

Why the test passed

The direct test was not wrong. It showed that tiling finds results when it runs. What it could not show was whether production ever gets to that point. That depends on the inputs the real job sends, and the real job always sends a box. A test that leaves out the box covers a different path than the one in production.

The rule

Run tiling on every frame, next to the box crop, and keep the best result. A box is a hint about one moment, not a boundary for the whole clip.

Test the path production actually takes, with the inputs production actually sends. A direct test without those inputs only proves the code works when it runs. It does not prove production ever runs it.