Tests measured a cache that was never cleared

On 20 August 2026 we made a commit to the monitoring app that fixed a problem that had sat in the integration suite for a while. Six of the eight tests for one lookup client were failing, and nobody had spotted it.

This post covers what was wrong, why one requirement could never have passed, and the two rules that came out of it.

What the tests were supposed to check

The monitoring app calls an external lookup service and caches the answers. The integration tests for that client run against a real Postgres database. They check how many HTTP hits the client makes and what it returns: an answer found in the cache, or an error.

Between tests, a helper resets the database. Each test is meant to start from an empty state, so a count like “exactly one HTTP request” means what it says.

What actually happened

The reset helper emptied fourteen tables. It did not empty the table where the lookup cache lives.

So every test started with the cache the previous test had left. The commit describes the result: “zero HTTP hits where the test expected one, and found from the cache where it expected error.” Tests that expected a network call got none, because the answer was already cached. Tests that expected a failed lookup got the earlier successful one.

One requirement was worse off than the rest. One item in the project’s definition of done rested on the assertion “no other HTTP hits”. The commit explains why that could never be checked: it cannot be measured when the first hit never happens. The commit says the item had been “asserted against a test that could not hold it”.

Why the table was left out

The omission had a reason, and that is what makes it worth writing about. Four tables were outside the reset list on purpose, the cache table among them. A project decision says events are a log and can be deleted, while knowledge is protected. In production, those four “knowledge” tables survive when events are deleted.

The commit names the confusion: that is “a property of retention, not of test isolation”. A rule about what to keep in production had leaked into the question of what to wipe between tests. The two have nothing to do with each other. A test database holds no knowledge worth keeping.

The only table still outside the list is settings, because it is seeded rather than written by tests.

A second test that could never pass

The same commit found a related bug. One test checks that a cached answer expires after a time window. Rather than wait thirty days, it backdates the cache row by forty days and expects a fresh fetch.

But it backdated a timestamp column on another table, and the cache reader doesn’t read that column. The cache had earlier moved into its own table. The test still aged the old column, so it could never have passed after the move. The doc comment above the seeding helper still described the old location too, and that was corrected in the same commit.

So the test file had two separate problems. The reset left stale state behind, and one test was setting up its condition in the wrong place.

How it was found, and the numbers

The commit gives before and after figures for the whole integration suite: 42 of 47 passing before, 49 of 49 after. The figures don’t line up exactly with “six of eight” in the one file, and the total went from 47 to 49. The commit doesn’t explain either difference, so we won’t guess at it here.

The build plan, under “Verified as of 2026-08-20”, lists 51 integration tests against a real Postgres with pgvector, all green. It adds that the lookup client’s test file “had six failing tests that nobody had seen”, and points to this commit. That is two more tests than the 49 in the commit, and the material doesn’t explain that difference either.

Henning writes no code himself, so the original reset list and the stale aging test were Claude’s. The material doesn’t say who first saw the failures. What the record shows is a test suite reported as passing while one file was mostly red, and a requirement asserted against a test that could not check it.

The guard that came with the fix

Adding the knowledge tables to the reset list created a new risk, and the commit deals with it directly. The reset now empties tables that production is meant to keep.

The reset function uses whatever database connection it is given. So the function now asks the database for its own name before it does anything. If the name is not the dedicated test database, it throws an error and empties nothing. The commit puts the cost at one query per run.

The fix and the guard belong together. Clearing more tables makes the tests honest. It also makes the reset more dangerous if it ever points at the wrong database, so the guard went in with the same change.

The rule

Clear every table that tests write to, between every test. Whether production keeps a table has no bearing on whether tests should reset it. The only exceptions are tables that are seeded, not written by tests, and each exception should be written down along with its reason.

And because a complete reset is destructive, the reset function has to check where it is pointed. It refuses to run unless the database is the test database.