Three weeks of a source delivering nothing

A source in the newsroom engine’s registry was switched on and delivered nothing for three weeks. While we tracked that down, we found a second source that had stopped working because of a compressed robots.txt. And the code that fixed both had been running in production before it existed in git.

A note on attribution first. Henning writes no code himself, so the original code behind these bugs was Claude’s.

The source that was active and empty

The newsroom engine collects news for kokkekunst.no from a registry of sources. Some publish RSS. Others only publish a sitemap, and many sites publish a sitemap index: a list that points to several child sitemaps. Following every child would pull in years of archive, so the collector follows just one.

The code comment said to follow the newest child. The code actually took the last entry in the list.

On WordPress sites running the Yoast plugin, the index lists posts first and authors last. So for one of our sources the collector read the author sitemap every time. It found no new articles. The changelog says the source had been active “since August” without delivering a single item. The commit message and the task log call it three weeks.

Nothing flagged it. From the outside, a source with no new articles looks the same as a source with nothing to say. We had seen this pattern before. An earlier changelog entry describes an RSS source that showed “0 new of 15” on every run because we built its links wrong, and it ends with: “A source that gives zero new every time is a symptom, not a quiet source.” This time we had the lesson written down and still missed the case.

How we found it

We didn’t find it by monitoring. On 2026-09-20 we did a session to expand the source registry. We ran twenty candidate sources through our feed-finding tool and added four. The ones we rejected are listed in the sources document with a reason, so the next session doesn’t try them again. The task log says both bugs were “found along the way”.

The fix for the index

The collector now reads each child’s lastmod and picks the child with the freshest date. If two dates tie, the one listed first wins, because Yoast puts posts at the top. If no child has a lastmod, the first one wins.

The rule has a limit, and the sources document says so. One site’s index has more than a hundred children with the same lastmod, so freshness can’t tell news apart from anything else. For that source the registry points straight at its news sitemap and skips the index.

The commit adds a regression test that copies the Yoast layout: four children, with posts first and authors last. The test expects the post sitemap. Further cases cover a list without dates (first wins) and an empty index (no choice).

The robots.txt that couldn’t be read

The collector respects robots.txt. It fetches the file once a day per host and caches it. If the file can’t be read, the source isn’t fetched either. In the sources document’s words: guessing is not respecting.

We ask hosts for uncompressed responses. Some hosts send gzip anyway (the code comment names nginx in front of WordPress/Yoast). The code that fetched robots.txt read the body directly as text, so the compressed bytes came through as text, null bytes included. Postgres refuses to store null bytes, so saving the cached file failed. The host then showed up as “could not read robots.txt” for about a day, per the commit.

The status message pointed the wrong way. “Could not read” is a bucket for several kinds of failure. According to the changelog, this one landed in the same bucket as a timeout. Nothing in the message pointed at compression or the database.

We had already met this problem once. Early in the build, the task log notes that the HTTP library “does not unpack gzip by itself” and that reading compressed bodies as text “looked like ‘unknown format’ and not like an error”. The main fetcher was fixed then.

The fix decompresses the body according to its content-encoding header. It also strips null bytes in every case, since a real robots.txt never contains them.

A third quiet zero

The same commit fixes one more silent failure. When a new source was verified, the verification step saved the feed’s ETag and Last-Modified headers. On the first real collection the host then answered “not modified”, so the source delivered nothing until its feed changed again. The commit’s comment says one of the new sources sat like that for a day. Verification no longer stores those headers.

Running in production, absent from git

The commit message opens with this: the work was done in the source registry session on 2026-09-20, “but was never committed. It has been in the working directory since, and runs in production.” The task log records the same thing when the branch was pushed: the source registry work “ran in production without existing in the git history”.

If the working directory had been reset or rebuilt, both fixes and four verified sources would have disappeared without anyone noticing. The bugs would have come back just as quietly as they arrived.

The rule

• Pick sitemap children by freshest lastmod, not by position. If dates tie, take the first. If a site’s dates can’t tell news apart from other pages, point the source straight at the news sitemap.
• Decompress by content-encoding, and strip null bytes from anything that gets stored as text.
• Commit the work in the same session. Code running in production must exist in git.
• Keep a regression test on the index choice, built on the real Yoast layout that broke it.
• A source that returns zero every time is a symptom. Look at it before you trust it.