The image backup was silently dead for seven weeks

For seven weeks, the photo backup for one of our apps produced nothing, and nothing told us. The container reported healthy every day. The database backup kept running, so the most important data was safe. But a review on 4 August 2026 found that those database dumps could not be restored into an empty database without a manual step. That step was not written down anywhere.

This post covers both problems, how we found them, and what we changed.

The setup

The app stores bottles, shelves, label scans, meals and vector embeddings in Postgres with the pgvector extension. It runs under Docker Compose. Two backup jobs write to a local backup folder:

• A Postgres dump every day at 03:30 UTC. It keeps 30 daily, 4 weekly and 6 monthly copies.
• An archive of the uploaded label and shelf photos every day at 03:45 UTC, kept for 30 days.

The photo job is a small shell script. Cron runs it inside an Alpine Linux container.

Hole 1: CRLF killed the photo backup

The shell script was checked out on Windows with CRLF line endings and mounted straight into the Alpine container. Alpine’s shell is busybox sh. It read the first line with a carriage return on the end, and failed with “illegal option -”.

So the script died on its first line every night. Cron wrote the error to a log file next to the archives, and nobody read that file. The container said healthy the whole time.

The photo backup stopped on 13 June. We found and fixed the problem on 4 August, so there were seven weeks with no new photo backups.

After the fix and a restart, a fresh archive appeared right away: 29.6 MB, 225 files.

Hole 2: dumps that could not be restored

The database side looked fine. A dump was written every day. The commit notes that the latest dump held 108 bottles, which matched the live database. The data was complete.

The dump job ran pg_dump with the --schema=public option. With that option, pg_dump leaves out the statements that create the vector and pgcrypto extensions. Without them, the restore fails before the first table with type "public.vector" does not exist.

You could get around this by creating the two extensions by hand before restoring. But you had to know to do that, and it was not written down anywhere. A real recovery, with the original disk gone, is a bad time to work that out.

The fix was to drop the --schema=public option. We checked it the same day with a full restore into a blank pgvector Postgres 17 container. All 108 bottles, 442 scan events and 108 embeddings came back with no manual steps. A comment in the compose file now says not to put the option back, and why.

Dumps made before 4 August still need the manual extension step. The new restore runbook says so.

How it was found

The commit calls these “two real holes found during review” of the backup setup.

Henning writes no code himself, so the script and the --schema=public dump option were both Claude’s. Both problems sat where two pieces met: a Windows checkout and a Linux shell, and a dump option and an empty database.

What we changed

Line endings are locked. A new .gitattributes file forces LF line endings for shell scripts and CRLF for Windows batch files, on any system that checks them out. The file has a comment about the 13 June outage, so the next person who wonders why the rule exists can read the reason.

Backup age is shown in the app. The app’s System page has a new backup check under storage. It reads the timestamps of the newest database dump and the newest photo archive and shows how old each one is. If either one is older than 8 days, the check turns to a warning. Both jobs run daily, so 8 days is generous: it allows for a weekly schedule plus some slack, and it can be changed. If there is no database dump at all, the check reports down, not just a warning. The app only has read access to the backups, so it can read file times but can never write to or delete them.

Ten new tests cover this check. One recreates the June to August case exactly: a fresh database dump and a 49-day-old photo archive, and the check has to say the photo backup has stopped. Another makes sure a freshly written log file in the same folder cannot hide an old archive. After the change, all 402 backend tests passed.

A restore runbook lives in the repository. The new document has tested restore commands for both the database and the photos. It describes both traps and notes that older dumps need the manual extension step. It sits in the docs folder rather than next to the backups, because git ignores the backup folder. As the commit puts it, the runbook has to exist even when the disk it describes is gone.

The rule

• A container that reports healthy does not prove the backup works. Check that the backup is fresh.
• Test the restore into an empty database. Until you have restored a dump, you don’t know that it works.
• If the repository is checked out on Windows and the scripts run on Linux, lock shell scripts to LF line endings with .gitattributes.
• Show backup age where people actually look. In the commit’s words, a backup nobody sees the status of is a backup you do not know whether you have.