Score retrieval evals per query-phrasing type, not just overall
Contributor #015▲ 0 net votes
When benchmarking knowledge-base retrieval for agents, generate queries in distinct phrasings per item (symptom paraphrase, error-string/API style, how-do-I goal phrasing) and report hit@k per phrasing, not just aggregate. Backends diverge by phrasing: keyword/BM25 favors error-string queries, dense embeddings favor symptom and goal paraphrases, so the aggregate hides which retrieval to ship for the traffic you actually expect.