The rulebook
Every rule below came out of something Claude got wrong while we built these apps. Each one links the post that shows what happened, how we found out and what changed. They are the rules we work by now.
They are short on purpose, so you can lift them into your own CLAUDE.md. The same list is at /rules.md as plain Markdown, and every mistake behind it is in an open dataset: JSON or CSV. Both areCC BY 4.0: use them freely, with a link back.
76 rules in 11 patterns.
A check that could never fail 11
A guard, watchdog, filter or test that looked like protection but could never trigger, or sat on a path the code no longer took.
Closest in the research: Wu (2026): Class E: operational omission and forensic blind spots
Check hiding rules against the real on-disk spelling: write covers with the real spelling, match excludes case-insensitively, and use ASCII mount names with a junction when needed.
From Windows treated Docs and docs as the same folder; the Linux container did not · vibeapp.win
A revision may add to the published record, never edit or remove it silently. The autopilot compares text, not the revision's own description, and pages with nothing new are skipped.
From The autopilot revised a page, dropped a published point, and its history note said nothing had changed · vibeapp.win
Decide duplicates by the commits a post cites, not by id or title: a published post citing more than half of a draft's commits makes it a duplicate, and publishing stops at two shared commits.
From The engine retold the same mistakes under new ids · vibeapp.win
Measure what the watchdog is meant to detect, test it against a known failure before trusting its green, treat silence as a symptom, and count a fault only when the system could have acted and didn't.
From Watchdogs that could never fire, or fired on normal operation · The stock app
A watchdog must measure the outcome, not the setup. Count subscribers and check that recent alerts went out. Feed that to fault tracking and show it somewhere that doesn't depend on the checked channel.
From The alert channel was dead for seven days while the status page stayed green · The stock app
Every filter or check needs a test showing it can fail, and
|| trueshould not be used by default.From A sitemap filter that could never filter · The kokkekunst.no site
A decision isn't a gate until code enforces it: freeze the exam outside training, make training refuse to run without it, include misses, and measure with the production code path.
From The gate was never code, and pointed the wrong way · The monitoring app
Verify a change alters behaviour in the running code, put guards in the one place every action passes through, and use a sensitivity row when a setting might never bind.
From Guards and dials that were built but never ran in the path that mattered · The stock app
Put the guard in the one function every order goes through, for both buys and sells, check our own order reference, count orders from the same cycle, and don't rely on the broker.
From The double-order guard sat in the legacy path while the whole fleet ran the target-portfolio engine · The stock app
Verify against markup only the tested thing can produce, such as the border-b-2 class, and count the elements (four buttons). A string that can appear elsewhere on the page proves nothing.
From My verification matched the page title, and I told the owner the tab bar was there · The stock app
If a check cannot fail on the errors you fear, it is not a check; CI now boots the server and tests real endpoints, and you should ask what change would turn a green step red.
From The CI check only checked syntax and could not catch a broken server · The gin guide
Failing silently 11
Something broke and reported success, or an error was swallowed into a normal-looking empty state.
Closest in the research: Columbia DAPLab: Exception & error handling; Wu (2026): Class C: error swallowing and dilution
Every failure shows as a banner on every admin page, raised with raise() and cleared with resolve(). A watchdog also checks outcomes every ten minutes, such as whether the post went out.
From A dead Instagram token cost two days of posts, and the only sign was one console line · vibeapp.win
Pick sitemap children by freshest lastmod, decompress by content-encoding and strip null bytes, commit work in the same session, and treat a source returning zero every time as a symptom.
From Three weeks of a source delivering nothing · The newsroom engine
Give every silent failure mode its own alert, check the model call log after each run, and make sure a test backs any comment about what should wake you.
From The auth failure that was recorded as a successful run · The newsroom engine
Build with visible errors, and before deciding a fix doesn't work, check that it is actually in the output.
From The fix that was never built · The newsroom engine · The kokkekunst.no site
Run against the real token first, treat success with an empty result as unverified, and change the design rather than widen the token.
From A REST design that could not work with a one-permission token · The newsroom engine
A test now checks the journal against the SQL files in both directions. After a migration, query for the thing it should have created.
From The migration that said "completed" and never ran · The stock app
Never show absence as all ok. Use a typed map over every verdict value so a new value breaks the build, and give abnormal states an exit code.
From Missing fields and empty results rendered as «all is normal» · The stock app
Healthy does not mean the backup works, so check backup freshness and test restores into an empty database. Lock shell scripts to LF in .gitattributes and show backup age in the app.
From The image backup was silently dead for seven weeks · The wine cellar app
Keep errors apart from empty results and throw when everything fails; a silent-zero sweep now flags all-zero columns unless registered with a reason and expiry date.
From Three errors that looked like empty results · The stock app
Don't use an
ascast to force a field into existence; add the field to the query's select and let the type system check it, so a missing field becomes a compile error.From A type cast hid a missing field, and every sale showed a dash · The stock app
For every constant that is a point in time, check it is in the past and whether a wrong value gives a wrong or a missing answer; if missing, add a guard, a visible health check and a failing test.
From A start-of-measurement timestamp set 15 hours in the future silenced all evidence · The stock app
Measuring the wrong thing 12
A number that answered a different question than the one asked, and made things look better or worse than they were.
A group flag must separate the group's own movement from signal quality, and stay quiet if results flip with signal direction. Flag text may only state what the system actually does.
From The self-audit kept telling us to fix Paris when the whole exchange was just down · The stock app
When a data source is shared between sites, filter every query on hostname (requestHost_in). Then check that the paths, referrers and hostnames in the results can exist on your site.
From The status page counted another site's traffic as kokkekunst.no's · The newsroom engine
Count totals from the source, here git, with method and date stored beside them. Hand-written page facts are only a fallback, and items with no number must show as missing, not drop out of the sum.
From The front page said "across 4 apps" because only four app pages had a number · vibeapp.win
Use the mean as the null for excess return and the median only for hit rate. Placebo and time-travel checks now run as standing audit guards: measure the measuring setup, not just the sources.
From Against the median, 52.5% of random sources looked like a proven edge · The stock app
Test every check on real data, match names as whole words with Unicode-aware lookaround, and treat noise in a check as a bug.
From Our quality gates flagged correct text, and a real error got through · The newsroom engine
When a labelling error is found in training data, check the exam for it and rebuild the exam from corrected rows. Mark the score break, since older regression and background scores no longer compare.
From The exam was poisoned by the same wrong boxes as the training set · The monitoring app
Run the detector on the full frame plus four overlapping quadrants, send empty results to a second check, and treat comments about what a model sees as unverified until measured.
From The comments promised enough pixels. The detector squeezed 4K down to 320x320 · The monitoring app
When you write that we "have" data, check the database, not the code. Where a rule can be made runnable, make it a check, such as the new six-check docs command run by a Stop hook.
From The docs claimed nine metrics when the database held two · The stock app
A row is not a decision: deduplicate on the decision (with the replacement instrument in the key) before counting. Show row count beside decision count, and add an audit check that fails if they drift.
From The exit measurement counted simulations, not decisions · The stock app
Group by decision and count distinct instruments. Use the same yardstick for every source. Verify a measure against a case where you know the answer.
From Measurements that answered the wrong question · The stock app
A baseline must be prequential, per tier, measured on the same rows as the model, and reported with the row count, or say there are too few rows.
From A baseline that made the model look 12 times better than it was · The lottery experiment
Check that a number means what its label says, including scale and definition, and compare numeric values as numbers, not strings.
From A badge that counted 55 bottles, and a 0–1 score shown as /5 · The wine cellar app
Written from memory 2
Names, facts or API details written from what the model believed instead of checked against the source.
Closest in the research: Wu (2026): Class D: chained hallucination and fabrication
Check every factual claim against the primary source, comments included. If the source says planned, write planned. If there is no source, leave the claim out.
From A calendar comment that said more than the sources did · The kokkekunst.no site
Take names from the source's own page, find feed URLs with a tool, and take version-specific facts from the installed version.
From Everything written from memory turned out wrong · The newsroom engine
Settings that did nothing 2
A setting, weight or input that existed in the code and the interface but never reached the decision.
Closest in the research: Columbia DAPLab: Codebase awareness & refactoring issues
Delete dead code and correct or mark the documentation as historical in the same change, so nobody debugs a component that does not run.
From Dead LLM ranking code that looked like it was running · The wine cellar app
Measure each dial's effect through the engine with impact, add a sensitivity row, and check that every code path passes the value.
From Dials and weights that never took effect · The stock app
Fine in the test, broken in production 10
Handled in a smoke test, a local run or one environment, and missing where it mattered.
Closest in the research: Wu (2026): Class A: environment and platform quirks
Test both Start and Stop on an app with a non-ASCII path before calling a Compose feature done. Turn bake off for such paths and use a config-only composeFiler list when the override is dev-only.
From Start failed where Stop worked: an ø in the folder name and a dev override · Portal
Synthetic time series in tests need real timestamps and a real calendar, with weekends and holidays. When you check a measurement, count it per day or per weekday: a gap that repeats on a fixed period is a calendar bug, not missing data.
From Every Wednesday was missing from the five-day measurement · The stock app
Check the built result, check that canonicals answer without redirect, and check shared defaults on the pages nobody opens.
From Every canonical URL pointed at a page that redirects · The kokkekunst.no site
When removing an article for an error, replace every field that repeats it, including the lead, and keep a regression test that checks this.
From The unpublished article still carried the error in its description · The newsroom engine
Clear every table tests write to, between every test, whatever production retains. Write down exceptions, and make the reset refuse to run unless the database is the test one.
From Tests measured a cache that was never cleared · The monitoring app
Run tiling on every frame next to the box crop and keep the best result. Test the path production takes, with the inputs production sends.
From Tiling worked in a test and never ran in production · The monitoring app
A known limitation handled only in the test tool is not handled, so the fix goes into the production path: an engine pause between orders and a 429 retry.
From Known rate limit handled only in the smoke test, not production · The stock app
Build and deploy the whole stack together, read new backend fields defensively as optional, and wrap the routes in an error boundary.
From Version skew gave a blank health page · The wine cellar app
Open connections with autocommit on, use explicit transactions for writes, and verify saved data through a separate connection set up the way production sets it up.
From Production writes were silently rolled back and the tests could not see it · The lottery experiment
Declare any package needed at import time explicitly with a comment saying why, and let a clean CI install judge whether the dependency list is complete.
From Green locally, red in CI: an undeclared transitive dependency · The wine cellar app
Data read the wrong way 8
Units, matches, merges or escaping that turned correct input into wrong data.
Closest in the research: Columbia DAPLab: Data management errors
Use concrete words, not short stems. Test each entry on words it should match and on names and everyday words it should leave alone. Write the list in the text's alphabet and keep one shared list.
From A stem match for «rus» turned a wine story sombre because of a name · The newsroom engine
Any text reaching a channel, generated or typed, goes through that channel's preparation step right before sending. Length limits are checked after it, on the exact outgoing text.
From Hand-edited captions skipped LinkedIn escaping · The newsroom engine
Write only the keys the caller actually sent, using settingsPatch() and not settingsSchema.partial(). Read the server's answer, not the component's state, before calling it saved.
From Every save reset every other setting · The monitoring app
Know what a timestamp records before you judge by it. Share the marker between the code that writes a row and the check that reads it, and show what a check leaves out instead of hiding it.
From Booking time read as event time: two false alarms about trading after hours · The stock app
The server copy is the source of truth at startup. Seed from local only when the server has nothing, never after a network error, and keep one shared sync hook.
From Sync merged the lists and brought back deleted items · The gin guide
Match text against keys using the longest key and word boundaries, and test short keys, substring keys and keys inside unrelated words.
From Brand matching picked the wrong brand, and matched inside words · The gin guide
When you add a new dimension such as variants, review every place that uses the old id as a key. Store the new level so old saved data still works.
From Marking one variant as owned marked the whole brand · The gin guide
Use every field to tell candidates apart and check its unit, set a confidence floor before linking, and don't fall back to name search when an existing link can't be fetched.
From Centiliters read as milliliters linked bottles to the wrong size · The wine cellar app
Words that stopped being true 3
A promise, label or message that was right when written and quietly became wrong when the system changed.
Alert text must describe what actually happened in that case. The alert function now takes an optional title, and a reused title must be checked against each new case.
From The alert you wake up for said the run had stopped, but it hadn't · The newsroom engine
Don't assume a static site is rebuilt daily. Compare the story date with today's date in Oslo time, and if they differ or can't be checked, show "Siste sak" with the date.
From 'Today's story' label showed a two-day-old story · The kokkekunst.no site
When a feature changes, search every text that promises something about it and update them together. Check absolute claims against what the site loads, and state the exception.
From The About page promised no third parties would see you. Then we turned on analytics. · The kokkekunst.no site
No plan B when a dependency fails 2
A call to a model, service or disk with no timeout or fallback, so one failure took the whole feature down.
Closest in the research: Columbia DAPLab: API & external service integration failures
Enrichment must never block. When it fails, degrade with degraded=True, tell the user, and renormalise the remaining weights so scores and thresholds keep their meaning.
From Pairing crashed with a 500 when the local model was down · The wine cellar app
Give every external call an explicit timeout, even local ones. Write files atomically: write to a temporary file, then rename it into place.
From Local LLM calls had no timeout, so one hung model could hang a request forever · The gin guide
The wrong diagnosis 8
A fix aimed at the obvious suspect, while the real cause was somewhere else.
What gets approved must be what gets sent: rebuild the Reel when the draft changes, hold a Reel with no script for a human, and allow quote cards only for text the story marks as a quote.
From The previewed Reel was not the Reel that shipped · The newsroom engine
Give every verb a time limit and kill the whole process tree. A lock counts only with a fresh heartbeat. Log errors instead of crashing. Portal reads the heartbeat and says when the guard is down.
From The job guard could look dead while it was alive · Portal
For a job tied to a time of day, don't use an interval timer. Tick every ten minutes, let the job check the clock against its slot, and post only once per day.
From An hourly timer counts from container start, so it could miss the 16:00 slot · vibeapp.win
Read the error text before trusting the status code, let the most specific signal decide, use 429 only as fallback, and add a test that names the case that went wrong.
From A 429 that meant "your quota is empty" was retried as if it meant "slow down" · vibeapp.win
Never mark a draft as given up for a reason that says nothing about its text. If quota or login fails, stop the run and resume three hours later from where it stopped.
From A usage-limit message was treated as a doubt about the text · vibeapp.win
Run heavy work in a child process and give long jobs their own queue. When an external failure repeats weekly, check our own job history for what ended then before blaming the supplier.
From The weekly search that put the whole API on idle priority, and blocked the broker login six Saturdays out of six · The stock app
Verify in the built result, not just in the source. Check the patched index.html in the build and fetch the real manifest over HTTP.
From Home-screen icon missing: the page icon was not the cause · The lottery experiment
Every serve must trace back to a producer source with a URL, because comparing with a secondary source, or finding it too poor to use, is not verification.
From Ten signature serves were wrong until we checked them against the producers' own sites · The gin guide
Other 7
Mistakes that do not fit the patterns above.
Give a model that writes one item at a time the recent history as data, and make the fallback rules avoid the same repeats.
From The director gave four of six Reels the same gold light · The newsroom engine
Refuse to write an already-written candidate, save marks under a lock on a freshly re-read list, and detect duplicates by shared commits rather than title, slug or id.
From Three parallel writers lost each other's marks, and the same mistakes were written up again · vibeapp.win
Write shared state under a lock and re-read the list right before saving, never from a copy loaded before slow work. Refuse to write a candidate that is already written.
From Three parallel writers overwrote each other's marks · vibeapp.win
Use one key per entity everywhere. A new page isn't done until it is linked in and indexed, which includes a check in URL Inspection.
From Person pages competed with tag pages, and plates differed per page · The kokkekunst.no site
Log the aiming decision at info, zoom only after a box was seen, stop when the box matches the last round, and wait for focus to settle with a ceiling instead of a fixed pause.
From The camera aimed twice at a stale box and zoomed blind · The monitoring app
Store the forecast that was actually used next to the prediction it fed, and always compare against a simple persistence baseline.
From Store the forecast that was actually used · The lottery experiment
Use placeholders, not examples, in prompt schemas. Add a deterministic echo-guard, prefer null to a guess, validate against known bottles, and check every prompt when one copies.
From Prompt examples the model copied word for word · The wine cellar app