Our quality gates flagged correct text, and a real error got through

The newsroom engine writes news for kokkekunst.no. Every draft goes through a set of gates before it can be published. One of them is the fact gate. It checks the proper names, numbers and dates in a draft against the source. When it flags something, the story goes to a queue instead of being published automatically.

In early September the gate was producing findings that were wrong. Because there were so many of them, a real finding went unnoticed.

The story that exposed it

On 2 September we published the same jury announcement for the national cooking championship (NM i kokkekunst) twice. The first version was based on the cooking association NKL’s own article from 31 August. The second was based on a trade publication’s article from 2 September. The changelog for version 0.3.1 records that the duplicate was also the worse version. It called NKL “Norges Kokkemesteres Landsforening”, a name the association stopped using in 2018. That name was not in the source at all.

The gates had done their job. According to the changelog, the fact gate flagged the outdated name word for word, and a separate model check noticed that the source itself used two different names for the same jury member. The story went to the queue anyway and was released from there. The 0.3.1 entry sums it up: the gates worked, and they were overruled.

The next release explains why they were overruled. In the jury story, the gate’s list of findings included “Landsforening” right next to the real problem. When a list mixes findings you have learned to ignore with ones that matter, the real ones get ignored too. The release notes put it this way: this is not theoretical noise.

How we found the cause

After the jury story, we measured the gates again on real drafts. The result was recorded in a code comment and in the changelog for 0.3.2, dated 4 September. Two false positives accounted for most of the noise.

The publisher’s own name. NKL’s articles never spell out “Norske Kokkers Landsforening”. They write “NKL” or nothing. Our source register holds the full name, checked against the association’s own front page, and the site prints exactly that text under each story as “etter X” (“based on X”). So whenever a draft named the publisher correctly, the gate flagged it, because the full name was not in the article text. In the measurement on 4 September, that happened in seven of fourteen stories, and it was always the same word.

In other words, the gate was treating our own verified register as suspicious.

Names matched as raw substrings. The source register includes a publisher called “NTB”, which is the Norwegian news agency. Known names were matched with a plain substring search on lowercased text. “restaurantbransjen” (“the restaurant industry”) contains the letters “ntb”, so the gate decided NTB was mentioned. It then flagged a proper name that appeared nowhere in the text. The 0.3.2 changelog notes that the register has many short names that do the same thing, such as “Smak” inside “smaker” (“tastes”).

Henning writes no code himself, so the original substring match was Claude’s. The material documents the design flaw and its effect: a check that was meant to catch invented names was inventing names itself.

The fix

The fix in version 0.3.2 has two parts.

First, the names and publishers of a story’s own sources now count as source text. The fact gate adds them to the text it compares against. The exception is narrow. It only covers sources the story actually cites. If the model writes in a publisher we do not cite, that is still flagged, and a separate test covers it.

Second, known names now only match as whole words. The new match uses lookaround: the name must not have a Unicode letter or digit directly before it or directly after it. It is case-insensitive and Unicode-aware, and names of two characters or fewer are still skipped.

Why not the usual word-boundary token \b? A comment in the code states the reason in one line: lookaround instead of \b, because \b does not know æ, ø and å.

The same release added a way to correct a draft in the queue from the command line. A correction creates a new draft version, and the gates run again. The old version is kept, so we can see what the model wrote and what a human changed. Before this, the queue could only publish or discard. That was part of the same problem: four finalist profiles were sitting in the queue with errors like “two words too many in a quote”. They were too small to discard and impossible to fix.

What it doesn’t fix

The gate still flags some ordinary words at the start of a sentence that do not appear in the source. An earlier measurement found these and chose to keep them, because removing them would also drop a real name that appears only once, at the start of a sentence. Some noise remains. The difference now is that we have measured it and know which kind it is.

The rule

Test every check on real data, not just on the cases it was designed for. Seven of fourteen was invisible until we ran the gate on real drafts.

Match names on word boundaries using lookaround with Unicode letter classes, because \b does not know æ, ø and å.

Noise in a check is a bug, not a cosmetic problem. A check that is wrong half the time teaches people to skip its findings, including the one that matters.