The auth failure that was recorded as a successful run

The newsroom engine writes the news for kokkekunst.no. It runs on a schedule, collects articles from a list of sources, asks Claude to sort and write, and sends anything uncertain to a review queue. When something goes wrong, it is supposed to send a push alert.

Before the fix, it handled one of its two most important failures and stayed silent on the other.

What the code promised

The alert module had a comment that said which case you should be woken up for. Translated from Norwegian: without a login or quota, the site fetches nothing, and it goes quiet in a way that looks like there just wasn’t any news.

That covers two failures: the model quota runs out, or the login to Claude stops working. From the outside, both look like a slow news day. The comment said exactly that.

The code handled only one of them.

Why quota alerted and auth did not

When the quota runs out, the model adapter throws a dedicated stop error. That error ends the whole run, and the pipeline catches it at the end and sends a “run stopped” alert. The reasoning in the pipeline’s header comment: once the quota is gone, hammering on just gets you a slower failure, and seven steps that all fail on the same thing.

Auth failures were deliberately not treated that way. A single call can fail on auth without the token being dead, so stopping the whole run on one auth error would be too blunt. That choice still holds.

But nothing replaced the stop. The commit that fixed it describes what happened instead: each model call failed by itself, the steps reported “0 stories”, the run was recorded as successful, and nothing said anything.

The task list made it worse in a quiet way. It had an item marked done: “alert when a run stops on quota or auth, same mechanism”. But a run never stopped on auth, so the alert for that case could never fire. Next to it was an unchecked item saying quota and auth errors showed up on the status page, but with no active alert out of the app. You would only see them if you opened the page to look.

What was already logged

Every model call is logged with an error class (such as rate limits, quota and auth), and each call is tied to the run step that made it. Before the fix, nothing checked that log for auth failures after a run.

The material does not record a specific outage date or how many runs were affected, so we won’t give a number. Henning writes no code himself, so the gap was in Claude’s code.

The fix

The commit is dated 20 September 2026. It adds a check that runs at the end of each pipeline run. It asks the log of model calls one question: did at least one call in this run fail on auth? If yes, it sends an alert.

Three design decisions are written into the code and the docs:

It does not stop the run. One auth error doesn’t prove the token is dead, so the rest of the run finishes as before, and the alert comes at the end.

It reads the call log, not an in-memory counter. The model calls happen deep inside the steps and are logged there. The log already knows. Copying that into a counter would be one more thing that could drift.

It swallows its own errors. The code comment calls it telemetry for an alert. If the query fails, the function returns “no auth failure” and the run carries on. An alert should never be able to break a run that otherwise went fine.

The alert says what to do: create a new token with claude setup-token and put it in place. According to the changelog, it has a different title from the quota alert: “the login is out of service” rather than “the run stopped”, because the run did not stop. It goes out on the same channel as the queue and quota alerts, and that channel is off until it is configured.

A new test file covers the check against a real test database. It fires when a call in the run failed on auth. It stays quiet when calls succeeded or failed on something else, such as rate limits or unknown errors. It counts only calls from its own run. It returns false instead of throwing for a run ID that matches nothing. The project’s definition of done gained a matching line: a run where at least one model call failed on auth must alert out of the app, even if the run otherwise completed.

The same shape, again

This was not the first silent failure of this kind in the project. The changelog describes a source that reported “ok” and “0 new of 15” on every run. Its feed links pointed to addresses that didn’t exist, so every item was discarded, and it looked like a source with no news. The rule written then was: a source that gives zero new items every time is a symptom, not a quiet source.

The auth failure follows the same pattern. Zero output plus a green status looks just like a calm day. If the only check is whether the run threw an error, you can’t tell them apart.

The rule that came out of it

• Every silent failure mode gets its own alert. If a failure doesn’t throw, it needs an explicit check, or nobody will hear about it.
• After a run, check the log of model calls, not just whether the run threw. The log already holds the answer.
• Alert code must never be able to break a run that otherwise went well. It catches its own errors and steps aside.
• If a comment says “this is the case you should be woken up for”, there has to be code and a test that does the waking.