The weekly search that put the whole API on idle priority, and blocked the broker login six Saturdays out of six

The stock app runs a strategy search every Saturday at 03:30. It backtests trading rules and parameters over three years of prices, and the work is synchronous: it takes about two hours and the CPU stays busy the whole time. Until 26 September 2026 it ran inside the API process, the same single Node thread that serves HTTP, runs the trading cycle, the scheduled jobs and the health checks.

This post covers two faults in that one job, found a week apart. The second was hiding behind a story we had written down as fact.

Saturday 26 September: a 6.1-second stall

That morning Henning asked whether everything was fine. The answer was no. At 04:54, in the middle of the search, the API’s event loop had stalled for 6.1 seconds.

The stall had an obvious explanation: two hours of synchronous work on the API’s only thread. The root cause turned out to be worse. Before starting, the search called Node’s os.setPriority with 0 and the low-priority constant. The purpose, according to the code comment, was to “yield to the app”. But 0 does not mean “the work I’m about to start”. It means the calling process. The search wasn’t running in a process of its own, so the call lowered the priority of the whole API.

We measured it the same morning. The API process sat at Windows priority 4 (Idle), and every other Node process on the machine was at 8 (normal). With Idle priority, the API gets CPU only when nothing else wants it.

Claude wrote that call. Each time the search ran, the API dropped to Idle and stayed there until the next restart. The commit’s notes say this happened every Saturday since August. Nobody saw it for weeks, for two reasons. Every deploy restarted the service and reset the priority, and no check ever read the priority.

Henning’s decision was short: give the search its own process. Claude moved the search into a separate child process with its own build entry. The parent starts it, forwards its log lines, reads the last JSON line on stdout as the result and kills it after a five-hour timeout. The search itself did not change: same triggers, same parameter grid, same seed, same filter. Only the child lowers its own priority now. A test checks that a child which lowers its priority leaves the parent’s priority unchanged.

We also added a guard for the general case, beyond this one call. A new health check flags the API process when it runs below normal priority, whatever set it. The project’s catalogue of fault classes got a new entry (number 21): a process-wide setting changed by a job running in the shared process. Priority is one example. Environment variables, the working directory, global agents and process-level handlers belong to the same family.

An accident during the fix

To smoke-test the new child process, we ran it with a deliberately invalid database URL. That should have failed fast. Instead it ran the real search, because the app’s environment loading replaced the fake value with the real configuration. We killed it after two minutes. It had written nothing, because the search only saves at the end. Since then, the docs say that a child which loads the app’s environment cannot be smoke-tested by overriding that environment.

Saturday 3 October: the broker break was ours

The broker’s login chain had broken on Saturday mornings for weeks. When the refresh token expires, someone has to log in again by hand. Our own operations docs called it the broker’s weekend maintenance, with the chain breaking every Saturday around 05:35. We had moved other broker-dependent weekend jobs to weekday evenings to work around it. The maintenance window was never measured.

On 3 October it broke again, and Henning asked Claude to look for faults and improvements, saying he was puzzled this one still wasn’t fixed. This time the job history answered it. No scheduled job ran between 03:30:51 and 05:49:51.

The app runs its 40+ scheduled jobs through one worker with a concurrency of 1. Most of them only put a job on their own queue and return. The strategy search did not. The worker waited for it inline, for about two hours. During that time nothing else on the schedule ran. That included the token renewal, which normally runs every ten minutes. On 3 October the last renewal ran at 03:20, and the refresh token expired at 04:19, about an hour later. The app only noticed when the scheduler woke up, which was when the search finished.

We compared the logged break times with the search’s end times:

Saturday	Break logged	Search finished
22 Aug	05:34	05:34:39
29 Aug	05:40	05:40:25
5 Sep	05:36	05:36:17
12 Sep	05:38	05:38:31
26 Sep	05:38	05:38:20
3 Oct	expired 04:19	05:49:51

Six out of six, to the minute. The first fix had put the search in its own process, but the scheduler still waited for that process to finish, so the queue stayed blocked.

Our monitoring missed this too. The fault-detection jobs run in the same blocked queue, so they stopped along with everything else. The fault register also had a side effect. Every hour it “fixed” the overdue token-renewal alert by registering the schedule again, when the real problem was that the broker connection was down. That alert is now suppressed while the broker-auth check is red.

The fix: the weekly schedule entry now only puts a job on a separate queue with its own worker, and returns.

What is proven and what isn’t

As of this writing, the queue fix is not proven. The docs name the evidence we expect: the first Saturday with the login chain intact, 10 October. We haven’t seen that yet.

The same session found two more faults in raw SQL. One was a date written in a format Postgres rejects; it failed on every exit, and a best-effort catch swallowed the error. The other was an integer that needed an explicit cast, which made the fund maturation step fail every evening. That is a separate story.

The rules that came out of it

• Heavy work does not run in the API process. Anything that computes for minutes runs in a child process, where process-wide settings only affect the child.
• A long job does not run inline in a serial queue. Anything over a few minutes gets its own queue. Fault class 22 includes a query that compares each job’s longest runtime against the list of inline jobs.
• When an “external” failure happens at the same time every week, check what in our own job history ended at that time before blaming the supplier.