A usage-limit message was treated as a doubt about the text

The engine that writes drafts for this site has an autopilot. According to our engine notes, it runs Monday and Thursday at 04:00 UTC and is checked every hour. It writes drafts and runs every draft through the gates (leak checks, fact checks, and checks that call the model). It schedules what comes out green twice in a row, one post per day. Anything else goes to the inbox for Henning.

The autopilot was set to start on 2026-10-04. The fix described here was committed that same morning. It deals with a case where the autopilot gave up on a draft for a reason that had nothing to do with the draft.

What happened

Some gates call Claude through the command-line tool. When a call fails, a small function reads the error text and sorts it into a kind: quota, auth, rate limit or unknown. The code comment explains why it has to read text: print mode has no distinct exit codes per error type.

Before the fix, the quota kind only matched messages about spending and credit: “spend limit”, “credit balance”, “insufficient credit”, “out of credit”. Those match pay-per-use accounts. We run on a subscription, and a subscription reports an empty quota in its own words. The commit lists examples like “Claude AI usage limit reached”, “You’ve hit your limit · resets 3pm” and “5-hour limit reached”. None of those matched, so they came back as unknown.

That had knock-on effects. The gate runner follows a deliberate rule, written in a code comment: a gate that could not run is not a pass, and doubt leads to the queue. So when a model gate failed for any reason, the runner recorded a “model gate failed” result with severity doubt. The message did not say why the gate failed.

The autopilot does not tell one kind of doubt from another. From its side, the draft had an unresolved doubt, and it gave up on the text. “Giving up” has a concrete meaning here. The autopilot stores a hash of the draft’s exact text and does not try that text again. The draft went to the inbox and stayed there.

The commit message puts it this way: the limit messages “were classed as unknown errors, so a gate that could not run became a doubt, the autopilot gave up on the text and it stayed in the inbox.”

The code was written by Claude, like all code in our repositories, so the misclassification is Claude’s mistake. The quota patterns had been written for one kind of account, and the subscription words for the same event differently.

How common this was

This was not a rare edge case. On 2026-10-03, the day before the fix, we measured the gates with a script that went through stored gate runs and counted the outcomes. One of its findings: 22% of model calls had hit the usage limit. The same measurement covered 269 leak-gate runs and 6,056 fact-gate claims. Before the fix, every draft that met an empty quota would have looked like a draft with a problem.

What changed

The code fix is in one commit, and a separate commit updates the engine docs to match. The code fix has four parts.

The limit messages are now quota. The classifier also matches “usage limit”, “hit your limit”, “limit reached”, “limit will reset” and “out of extra usage”. A code comment gives the reason it matters: retrying within minutes only fails slower.

The gate result says why it failed. When a model gate cannot run, its message now includes the error kind, for example “could not run (quota)”. A code comment says the point is to let the autopilot tell an empty quota from a bad answer. The finding is still a doubt, so the rule that a gate that did not run is not a pass still holds.

Quota or auth stops the whole run. Every gate call in the autopilot now goes through a wrapper. If a model gate failed with kind quota or auth, the wrapper throws, and the run stops. The autopilot already had a resume mechanism: if a run fails, it tries again three hours later and continues from where it stopped. A quota stop now uses that same path. Stored gate reports where a model gate had failed are no longer reused as if they were fresh, so the next run checks the draft again.

Some reasons never count against a draft. The autopilot now has a short list of reasons that say nothing about the text: a model gate that failed, model gates that have not run, and a fix attempt that failed. If any of those is among the reasons a draft did not go green, the autopilot still reports it in the inbox but does not store the “gave up” hash. The next run will try the draft again. This applies to posts and to app-page revisions alike.

The docs also note that the parts of the autopilot that do not use the model, the planner and the Instagram step, run regardless.

What we took from it

The gate runner’s caution was correct. A check that did not run should never count as a pass. The mistake came one step later, when the autopilot read “did not run” as “found a problem”. These are different statements. One is about the infrastructure and the other is about the text, and only the second should change what happens to a draft.

The second lesson is narrower. Error classification by text matching only covers the messages you have seen. The patterns were written for one way of paying, and the subscription words the same event differently. Since the fallback was “unknown”, and “unknown” fed into “doubt”, missing a pattern turned into a judgement about the draft.

The rule

A draft is never marked as given up for a reason that says nothing about its text. If a gate could not run because the quota is empty or the login is gone, the run stops and resumes later from where it stopped. The draft is not the problem.