Dead LLM ranking code that looked like it was running

We have a wine cellar app. Given a meal description, it recommends wines from the cellar. The code and the documentation both suggested that a language model helped choose that bottle. It never did.

On 2026-08-12 we deleted the module that was supposed to do this, along with its tests, schemas and cache keys. We also rewrote the documentation that still described it. This post covers what was there, how it misled us and the rule we took from it.

What the docs said

The feature document for wine pairing described a hybrid ranking with fixed weights:

• 0.40 × embedding similarity
• 0.30 × LLM rerank
• 0.20 × classic pairing rule
• 0.10 × drink-window fit

The ranking module held the full ranking prompt and named a separate prompt document as its source of truth. In short, the prompt asked the model to choose a top pick, score it and explain the choice.

The code matched that description. A module built the prompt from the candidate wines, parsed the JSON reply into typed schemas, clamped stray scores and cleaned up the text. There was a cache key namespace for LLM answers, with a prompt version string to invalidate old ones. There were nine unit tests.

What actually ran

The function that would have sent the request to the model was defined in the recommendations service and was never called. Nothing in the recommendation flow reached it. The commit message says the ranking had been fully deterministic all along. Git shows something slightly different: the first version, committed on 14 May 2026, did call the model. A change six hours later the same day took the model out of the decision and switched to a deterministic ranking, but left the call function, the module and the docs in place. From then on the ranking was deterministic. A deterministic ranking function in the sommelier service did the work, with the drink window as its own factor. The 0.30 LLM weight in the docs matched nothing in the running system.

A language model does run in the pairing flow, but it plays a different role. Its prompt calls it the polish layer. It gets the wine that has already been chosen and rewrites only the explanation, asynchronously in a background job. The updated prompt doc states that it cannot change the wine or the score. It writes the prose and has no effect on which wine wins.

What it cost

The commit gives the reason for deleting the dead path rather than leaving it alone: the code looked as if it was running and had already misled a debugging session. The material doesn’t describe that debugging session, so we won’t guess at the details.

The module had nine tests of its own. Green tests on dead code only show that the dead code is internally consistent.

Who got it wrong

Henning writes no code himself, so the unused module, the leftover call function and the docs were Claude’s, and so was leaving them in place when the model was taken out of the ranking. The code and the docs drifted apart from the running system, and nothing in our process noticed.

What the cleanup removed

The commit deleted the whole subtree:

• the ranking module itself (the prompt, the message builder, the response parser and two helper functions)
• the unused model-call function, the cache-key helper and the cache constants in the recommendations service. The commit notes that only the dead path ever wrote to that cache namespace.
• the two schemas for the LLM’s raw output and its rerank result
• the test file, with its nine tests of dead code

The test count shows the subtree really was isolated. Before the change there were 467 tests. After it, 458 passed, which is exactly 467 minus the nine deleted ones. Nothing else broke, and the linter and type checker were clean. If any live path had depended on the module, something outside its own tests would have failed.

Fixing the documentation too

The commit message says the documentation was corrected, not just the code. The top-level feature doc still claimed the ranking was hybrid with 0.30 × LLM, and deleting the code would have made that claim even harder to trace.

So in the same commit:

• The prompt document is now marked historical, with a status note at the top saying the prompt does not run and pointing to the polish prompt as the LLM that actually runs. We kept the original text as a reference in case we ever want real LLM ranking.
• The feature document got a dated correction block at the top. It says the hybrid weights describe the original design, not what runs, and that the ranking is fully deterministic.
• Doc comments in two other service files that pointed at the deleted prompt were updated.

We didn’t delete the old design documents. We labelled them, so a reader can still see what was planned without mistaking it for what exists.

The rule

Delete dead code and correct the documentation in the same change. Code that is defined but never called looks like it runs. Documentation that describes something that doesn’t run is worse than no documentation, because it sends the next person, human or model, to debug a component that isn’t there. When a design is dropped or never wired in, either remove it or mark it clearly as historical, with a date and a pointer to what actually runs.