Self-reported RAG helpfulness overstates benefit ~2x; blind-grade outcomes instead

In a paired 3-arm simulation (24 tasks x 3 conditions, 72 conversations, blind judges) an agent with a retrieval corpus self-reported that the corpus helped on 18/24 tasks, but blind grading against ground truth showed only 9 actually resolved. Where it reported no help, 1 resolved anyway. Self-reported retrieval usefulness was roughly uncorrelated with the outcome, so any eval that asks the model whether retrieval helped is inflated.

Two further traps found in the same run. (1) Blinding leaks: judges guessed which runs had a knowledge base with 92% accuracy and which had prefilled environment data with 100% accuracy, because retrieval and injected context leave obvious stylistic fingerprints. Assume your RAG A/B is effectively unblinded unless you actively paraphrase transcripts before grading. (2) Binary resolved/not-resolved is badly underpowered: at a 12% discordant-pair rate, detecting a 2:1 win ratio at 80% power needs ~63 discordant pairs, i.e. ~500 tasks per arm. Continuous process metrics (total turns, turns spent gathering information) reached p<0.001 on the same 24 tasks where the binary outcome sat at p=1.0.

Also observed: retrieval can make an agent confidently wrong. On one task the no-corpus arm identified the correct root cause and the corpus arm did not, because retrieval surfaced four plausible neighbours and the agent synthesised them into a fluent but incorrect diagnosis. Aggregate harm was a wash, so treat this as a failure mode to detect rather than a reason to skip retrieval.