The docs claimed nine metrics when the database held two

On 2 August 2026 a document in the stock app said the historical fundamentals table, going back to 2009, held nine accounting metrics. Claude had written that document earlier the same day. The database held two: diluted EPS and revenue.

The number was not invented. It came from the code. That day Claude had expanded the mapping that pulls figures from SEC filings, and the expanded code would produce nine metrics. But the weekly job that runs that code had already run at 01:00 that morning, hours before the expansion. The nine metrics would not arrive until the next run, on Sunday 9 August.

The commit message that fixed it puts it plainly (translated from Norwegian): “I described what the CODE will produce as if it were what the DATABASE contains.” It also names the cost it avoided. The mix-up “would have been discovered in 30 days when someone tried to backtest on data that did not exist.”

How it was found

Henning asked for a verification of the verification. The app already had a document describing how its own numbers are checked, and a second one describing how much history each data source has. Henning’s request was to test those documents themselves.

Claude took every claim in the two documents that could be checked and ran each one against the database. Round one: 21 claims, 20 correct, 1 wrong. The wrong one was the nine metrics.

After the fix the commit lists more rounds. Round two ran the whole verification recipe exactly as written: 12 of 12 invariants held. Round three tested code written earlier that day that had not been run yet. Round 3b re-tested the corrected claims: 6 of 6 held. The test suite stood at 695 green tests.

What the search for the nine metrics turned up

Working out when the nine metrics would land showed they would never cover all the stocks. The function that syncs SEC data picked 60 US instruments a week, with no sort order. Without a sort, Postgres returned roughly the same 60 rows every week, so the rest were never fetched. On 2 August Claude measured 343 active US instruments, of which 76 had fundamentals, and that number had stopped growing.

It was worse than that. Instruments with no SEC company ID, such as ETFs like SPY and foreign companies like ASML, were near the top of the unsorted list. They took up slots in the 60 and produced nothing, because they were only skipped after the limit had been applied.

The fix filters out instruments without an ID before the limit, sorts so that never-fetched instruments go first, and raises the limit from 60 to 120. According to the commit, coverage should now work its way through all the matched instruments over a few weeks instead of staying at 76.

The same verification caught a smaller gap. A new measurement of dead data series had been added to the morning report’s data that day, but the function that renders the report never printed it. The commit compares this to an earlier bug in the same app, where a guard calculated the right answer and the answer never reached anyone.

The corrected document

The fix did not just change “9” to “2”. Claude added a new chapter to the data history document called “What is NOT there yet”, with a table of claims next to what was actually true that day:

• Nine SEC metrics: the database has two. The nine arrive after the weekly run on 9 August.
• Fundamentals for the whole universe: 76 of 343 US instruments.
• The new daily metrics table: one day of data, useful as a backtest window around November at the earliest.
• 41 new global market series: written from the first sync after deploy, with no history going back.

The chapter states the rule: when you write that we “have” data, check the database, not the code. The commit adds that a count query takes two seconds, and that this is “the difference between an answer key and an expectation.”

Not the first time

Five days later, on 7 August, the commit that made documentation a measured duty listed the same kind of error showing up earlier in other places. “We have paid for that difference four times”. Among them:

• A go-live criterion said “lock missing” for two days after the lock existed.
• A list of untested code said a portfolio variant was untested while it was running.
• Three files gave three different counts of scheduled jobs.

The commit calls them all “hand-set flags in prose.” A sentence saying the documents “should be kept up to date” is the same kind of thing: it can drift from reality, and nothing that runs can correct it.

The trigger that day was a new case of drift. Fourteen commits over 6 and 7 August were missing from the progress log, the document that calls itself the record of what has been done. The main project file still said “last updated 2026-08-05”, after the fund module and several other features had been built and deployed. A new Claude Code session reading those files would have read about an app that no longer existed. Henning asked whether the docs could be kept current without him having to remind anyone.

The documentation duty became a check

Claude built a command that runs six checks against files and git and exits with an error on drift:

• the progress log covers every code commit
• the project file’s “last updated” date is no older than the newest code commit
• every document is listed in the project file’s reference list
• no document links to a document that does not exist
• every command mentioned in the docs exists as a script
• the job count in the jobs document matches the scheduler

Commits that only touch documentation do not count as drift. Without that filter, updating the progress log would itself create drift, and the check could never pass. The commit’s reasoning: “a guard that cannot turn green gets switched off.”

The first run found two real errors. One document described a command that existed as a file but not as a runnable script. Three documents were missing from the reference list, so no new session had ever been pointed to them.

Enforcement has two layers, and neither blocks anything. The rule sits in the project file that every new session reads. The check also runs as a Claude Code Stop hook at the end of each turn. The hook stays quiet when everything passes, never blocks, and says nothing if the check itself fails. It is deliberately not a git hook, because that would block commits in the middle of work where the documentation naturally comes last.

One limit should be stated. The check reads only files and git, never the database. It would not have caught the nine metrics. A claim about what the data contains is still covered by the written rule, not by the script.

The rule

If you write that we “have” data, check the database, not the code. Code shows what will exist. The database shows what exists now. Where a rule can be made runnable, make it runnable, because a rule that only lives in prose is just another claim that can drift.