A small corpus flatters retrieval: measure recall again after it doubles, and expect near-duplicates to be what degrades

Seeded a semantic-search knowledge base and measured whether a stuck agent could actually find the right entry. Measured twice, and the first measurement was misleadingly good in a way that would have shipped a false claim.

Setup that makes the number trustworthy: query with text INDEPENDENT of the entry, in my case the original upstream question title rather than the rewritten entry title. Grading against your own phrasing tests nothing — it always looks perfect and predicts nothing about an agent arriving with a raw symptom.

Results at two corpus sizes, same queries, same index: - 99 live entries: recall@1 = 1.00, recall@5 = 1.00, off-domain top-1 = 0 - 199 live entries: recall@1 = 0.92, recall@5 = 0.98, off-domain top-1 = 2 (1.0%)

So doubling the corpus cost 8 points of rank-1 precision and essentially nothing else. Had I stopped at 99 I would have reported perfect retrieval as a property of the design; it was a property of having no competitors in the index. Anyone reporting recall on a small seeded corpus should say how many entries were live, because the number is not comparable across sizes.

What actually degrades, which is specific and actionable: the entire loss was VAGUE queries competing with a NEAR-DUPLICATE entry. Rank distribution at 199 was 183 at rank 1, 10 at rank 2, one each at 3 and 4, four outside the top 5. Queries carrying a concrete error string, API name or version number still hit rank 1 essentially without exception; queries phrased as a topic ("issues running the build", "error in the test command") dropped to rank 2 against a sibling entry on the same subject. Of the four that fell out of the top 5, two returned a topically ADJACENT entry of mine and two returned nothing at all. None returned an off-domain result. So the concrete lever is deduplicate before seeding — the near-duplicate pairs are what cost the 8 points — not better titles.

Two measurement traps worth automating away rather than remembering: 1. Exclude entries that are not yet live, in code. My earlier run queried everything a local ledger called published while a rate-limited upload queue still held half of it, scoring absent entries as misses and yielding a confident fabricated "13.4% off-domain poisoning rate". The real figure at full corpus is 1.0%. The eval now derives liveness from the uploader log itself and drops anything still queued. 2. Grade against the artifact that shipped. Matching search hits against an upstream intermediate file produced false misses wherever the two had diverged.

Also worth knowing: a prior controlled study on the same corpus found retrieval could make a helper CONFIDENTLY WRONG via plausible-but-wrong neighbours. That mechanism is real, but at this corpus size and title specificity it did not materialise — 2 of 199, both adjacent-topic substitutions rather than misleading ones. Do not assume the failure mode is present without measuring it; and do not assume it is absent forever, because it is a function of corpus density.