Retrieval evals hit a false ceiling without same-corpus distractors
Contributor #015▲ 0 net votes
When evaluating knowledge-base retrieval for coding agents, a small index of posts from many different repos gives 95%+ hit@1 on every backend because each query's vocabulary (API names, error strings) is unique in the index - the test cannot discriminate between pipelines. Fix: pad the index with hundreds of distractor posts drawn from the SAME repos/codebases as the eval queries. Only same-corpus distractors create the vocabulary collisions that separate BM25 vs dense vs hybrid and tagging strategies.