The job guard could look dead while it was alive

Portal can start, stop and restart the apps in the park. Instead of running Docker itself, it hands a job request to a small Node script on the host, the job guard, which runs one of four fixed verbs through docker compose: start, stop, restart and fetch logs. Portal then polls for the result and shows a status line under the button.

Claude wrote the guard, as it writes all the code in these repositories. On 7 October 2026 a commit went in whose subject line says what had been wrong (translated from Norwegian):

Job guard: can no longer look dead while it is alive

What we saw

Stop sometimes hung. The status line read “waiting for the guard” and kept counting.

The guard had died before, and that had been handled separately. It had been killed twice from outside without the machine restarting, and then stayed dead until the next login, so Start and Stop put jobs in a queue nobody read. The fix for that was an extra trigger that starts the guard every five minutes, on top of at login. If the guard is already running, the extra start does nothing. The comment in the setup script explains why the built-in restart setting was not enough: it only covers a task that fails to start, not one that dies while running.

This time the guard was not dead. The session notes for 7 October say the guard was alive, with a process ID from 4 October. Debugging found three separate holes, and all three looked exactly like a dead guard from Portal’s side.

Hole 1: a hung docker compose blocked the queue forever

The guard runs jobs one at a time. It started docker compose and waited for it to exit, with no upper limit. A code comment added in the fix lists the realistic causes: Docker Desktop not answering, or a container that will not stop. When that happened, the guard kept waiting, and every job behind it stayed in the queue. Portal kept saying “waiting for the guard”.

The fix gives every verb a time limit: 20 minutes to build and start, 3 minutes to stop or restart, 1 minute to fetch logs. When the limit runs out, the guard writes a line to the job log saying it gave up and asking whether Docker Desktop is answering, marks the job as failed, and moves on to the next one.

Killing the command turned out to need its own fix. On Windows, docker compose starts its own child processes, and a plain kill only takes the top one. The rest stay alive and keep holding the app folder. The guard now kills the whole process tree.

Portal’s own patience changed to match. The frontend used to stop following a job after 600 seconds, which is shorter than a 20-minute build. It now waits 1,300 seconds, a bit longer than the guard’s longest limit.

Hole 2: the lock was fooled by reused process IDs

Only one guard may run at a time, so the guard keeps a lock file with its process ID. On startup, a new guard read that ID and asked the operating system whether such a process existed. If it did, the new guard assumed another guard was running and exited.

That check is not enough on Windows, which reuses process IDs. The changelog describes what went wrong: a guard that had been killed left its lock file behind. If Windows then gave that ID to some unrelated process, every new guard concluded the old one was alive and shut itself down.

The fix is a heartbeat. The guard writes a small pulse file every five seconds, even in the middle of a build. The lock now counts only if the process exists and the pulse is fresh, meaning under 30 seconds old. Otherwise the new guard clears the lock and logs why: the process exists but has not sent a heartbeat, so it is not a guard. The guard also takes the lock before its first round, so a second guard cannot grab jobs from the one that is already running.

Hole 3: an unexpected error was a crash

Some errors were not caught. The file watcher that notices new jobs could also fail along the way, for example if the folder is deleted and recreated. In Node, an error event with no listener crashes the process. The comment in the fix puts it plainly: an error with no listener is a crash.

Now each job runs inside its own error handler, so a failure is logged against that job and the loop continues. If the file watcher drops out, it is set up again. Until it is back, the guard’s regular timed rounds keep taking jobs.

Making “dead” visible

The three holes had one thing in common. From the outside, a busy guard, a guard locked out by a stranger’s process ID and a crashed guard all looked the same. Portal could only count seconds.

Portal now reads the pulse. If it is older than 30 seconds, the job status says outright that the guard is down and gives a hint, instead of counting. The job stays in the queue and Portal keeps watching, so it runs as soon as the guard comes back. The status command in the setup script also shows the heartbeat’s age, because, as its comment says, the scheduled task can show “Running” while the Node process under it is gone. The pulse is something the guard writes itself, so it tells the truth.

A smaller fix came along in the same commit. The guard’s log was piped through PowerShell 5.1’s *>> operator, which writes UTF-16, so the log became unreadable after the first restart. The guard now writes its own log as UTF-8.

What was left for Henning

The session notes list two steps only Henning could do. Portal’s container had to be restarted before the new status text would show. The scheduled task also had to be registered again from his own PowerShell window before the log would switch to UTF-8. The new guard itself was already running on 7 October.

The rule

• Every verb has a time limit, and the whole process tree is killed when it runs out.
• A lock counts only with a fresh heartbeat, not because a process ID exists.
• Unexpected errors are logged, not fatal, and the file watcher is set up again if it drops out.
• Portal reads the heartbeat and says so when the guard is down, instead of counting seconds.