Watchdogs that could never fire, or fired on normal operation

The stock app trades on its own, so it has a set of watchdogs: small checks that report to a status page and a fault register, so nobody has to look at the app every day. Between late July and the end of September 2026, several of those checks turned out to be blind to the failures they were built for. Another one cried wolf about ordinary operation.

None of these were found by the checks themselves. A real outage, a direct question from Henning and a missed news event exposed them. The quotes below are translated from the Norwegian commit messages and docs.

The tunnel check that answered “ok” during an outage

On 2026-07-31 the Docker Desktop engine died. It returned a 500 error even though its processes were still running. The Cloudflare tunnel connector is the only part of the stock app that runs in Docker, so the app could not be reached from outside. Everything that runs natively kept answering 200.

That same day we had built a check for exactly this case: can the app be reached from outside? During the outage it reported:

status “ok”, HTTP 302, “responds 302 in 71 ms”

The check treated any 3xx as “Cloudflare Access steps in before the origin, so the route works”. The commit that fixed it explains why that is wrong: Access is evaluated at the edge, before the origin is ever fetched. An unauthenticated request never reaches the tunnel connector, so the response is the same whether the tunnel is alive or dead. In the commit’s words: “The check could NEVER have seen a dead tunnel.”

The commit message also records that the same wrong conclusion was drawn live during the session: “I made the same faulty inference myself in this session: concluded ‘the tunnel is up (302 in 54 ms)’ before I tested whether anything reached the origin at all.”

The fix added a new status. A redirect to Cloudflare Access’s login domain now shows as “access-gated”: yellow, not green, with the text “the tunnel’s state is UNKNOWN”. If an Access service token is configured, the check sends it, gets through to the origin and can give a real ok or tunnel-down answer again. A 530 still means tunnel-down. A plain redirect that does not point to Access is still the app’s own redirect, and counts as ok. The commit added four tests, bringing the suite to 653 green tests. Recovery turned out to need a manual step too: Docker Desktop kept reattaching to the dead processes until they were killed first. After that the engine answered in about 10 seconds.

The notification channel that was green for seven days

On 2026-09-19 at 06:00 the push service answered HTTP 410: the device had dropped or rotated its subscription. The server deleted the only subscription, which is what the spec says to do. After that, nothing subscribed again.

Over the next seven days, 44 notifications reached nobody, according to the delivery flags in the app’s notification table. Two of them, on two Saturdays in a row, said the broker login was broken. Before 19.09, push had been delivered every day.

The status check for the notification channel stayed green the whole time. It read configuration (were the push keys present?) and never looked at delivery. The fault register also skipped this check on purpose. And the only channel that could have reported push was dead was push itself.

It came to light on 2026-09-26, when Henning asked us to look at the notification channel. The first measurement with the new check said “stale, 44 undelivered”.

The fix changed what the check measures. It now counts subscribers and reads the delivery flags on the latest notifications. Three undelivered in a row with subscribers present, or push configured with zero subscribers, turns it red. The check is now part of the fault register. The app’s bell icon blinks “!” when push reaches no device, so the app itself works as a second channel. The service worker now resubscribes when the browser rotates a subscription, and every visit quietly resubscribes if permission is already granted. One gap is written down as unsolved: there is still no independent channel, so a day with no app visit and no push is still blind.

The portfolio watch that guarded an empty table

The portfolio watch looks for negative news events on instruments you hold or follow.

It built “held” only from the table of manual transactions. But the app trades on its own, and Henning never trades manually, so that table had 0 rows. The watchlist table also had 0 rows. The watch’s condition could never match. It reported 0 hits in 27 of 27 runs, which looked like “no negative news”, not like a watch that could see nothing.

On 2026-08-01 an earnings-miss event with materiality 0.50 came in for a live holding. Nothing fired. The fix on 2026-08-02 reads holdings from the live simulation, which is the app’s actual portfolio, and keeps the manual table in the union. We verified it by running the fixed code inside a database transaction that was rolled back afterwards. The watch caught the event and produced a high-severity insight, and the database was left unchanged.

The other direction: alarms on normal operation

A check that shouts at nothing is also broken. The “LIVE is standing still” alarm fired on 17.09, 20.09, 28.09 and 30.09. Each time, the portfolio was full or the cycle had already traded. The fourth one came from a thinly traded Oslo stock that often has no trade before about 10:00, while the portfolio held 10 of 10 positions. The safeguard waited correctly. The fault register read a skipped candidate as a blockage.

The rule now: a skipped candidate only counts when there was a free slot and the cycle placed no order. Henning’s reaction the same day was that “the app should just be able to fix things like this itself”. Reasons with a clear, safe remedy now get an automatic fix (re-queue the quote sync, or run the reconciliation) and are only pushed if two attempts fail.

Two days earlier a similar thing showed up in drift monitoring. A fixed drift threshold of 0.25 flagged normal daily variation as drift: between 17 and 100 percent of the training days were over it themselves. The threshold is now calibrated per feature.

The rule

• Measure what the watchdog is meant to detect. Delivery, not configuration. Reaching the origin, not getting an answer from the edge. The actual portfolio, not a table that happens to exist.
• Test the watchdog against a known failure before trusting its green. If it cannot tell a dead tunnel from a live one, it is reporting “don’t know” as “ok”, and it should say “don’t know”.
• Silence is a symptom. Zero hits in 27 of 27 runs is a measurement only if the check could have hit anything at all.
• Overcorrection is also a failure. A watchdog that fires on normal operation teaches people to ignore it. Count something as a fault only when the system could have acted and didn’t.