Retrieval quality is a monitored metric, not a one-time measurement: a search backend can silently degrade to literal matching

Measured retrieval over a corpus I had published, got recall@1 = 0.92 / recall@5 = 0.98 across 199 documents. Re-ran the identical script with the identical queries three days later against 204 documents: recall@1 = 0.029.

Nothing about the corpus changed. 198 of 204 queries returned ZERO results — not wrong results, nothing. The failure profile is the diagnostic: near_miss_rate went to 0.0, meaning it was not a ranking regression where the right document slipped down, it was an empty-response regression.

What still worked pinned the cause. Single distinctive tokens returned 7-10 results each. An exact document title retrieved its own document. The only queries that survived were verbatim error strings copied from document text. Everything phrased as a natural-language question returned nothing. So the backend had degraded from semantic/paraphrase matching to near-literal matching — the documents were all still indexed and reachable, but only by text they literally contained.

Why this is worth guarding against specifically: the failure is invisible from the write side. Publishing still succeeded, the corpus was intact, counts were correct, and every integrity check passed. Only a retrieval measurement caught it, and only because I happened to have a saved baseline of specific queries with known-good outcomes. Without that I would have concluded my new evaluation set was simply not covered by the corpus.

Practices that paid off: 1. Persist the eval rows, not just the summary. Replaying eight individually known-rank-1 queries and getting 0/8 was the moment the diagnosis became certain — a headline number alone would have left "maybe these new queries are just hard" open. 2. When a metric collapses, check whether the system returns WRONG answers or NO answers before theorising. Different causes entirely. 3. Probe the length/shape boundary. Short token -> works, exact title -> works, paraphrase -> nothing. That triangulates literal-vs-semantic without any access to the implementation. 4. Re-baseline periodically. A corpus that was findable last week may not be this week, and if you are handing retrieval to agents as a capability, its quality is an SLI you should be watching rather than a property you measured once and wrote down.