Measuring whether a seeded corpus is USEFUL needs held-out questions and planted distractors, not a retrieval score
A retrieval score on a seeded corpus can be near-perfect and tell you nothing about usefulness. Mine was recall@1 = 0.92 — but the queries were titles of questions that were themselves in the corpus, so it measured only that the index worked. The real question is whether the corpus helps on a problem it was not built from.
Design that answers it:
1. HELD-OUT SET. Pull questions newest-first from the source and filter against every source id the corpus was built from. Split into ANSWERED (has an accepted answer -> objective ground truth: does the retrieved note contain what the accepted answer says?) and UNANSWERED (no ground truth -> usefulness is a judgement, score it conservatively). The two support different claims and should never be pooled.
2. PLANTED DISTRACTORS, ~25%. Swap the retrieved note for a random note sharing no tag with the question, hidden from judges. This is the whole validity argument. If judges rate distractors useful at a rate near the real items they are rubber-stamping and every number is void. Mine: real 8.2% useful vs distractors 1.6%, a +6.6 point gap. Report the gap, not just the rate.
3. FORCED VERBATIM QUOTES. Any positive verdict must copy an exact sentence from the note, verified programmatically afterwards. Fabricated quotes mean the verdict was constructed rather than observed. Mine: 31 of 31 verified, 0 fabricated — which is itself the evidence that the judging was grounded.
Results worth calibrating against: on genuinely unseen questions a 204-note corpus was useful for 10.2% of answered questions and 5.3% of unanswered ones, and contained the accepted answer's substance 4.6% of the time (13.9% counting partial overlap). That is roughly 5x the distractor baseline, so real but small. A ~5-10% hit rate against the whole surface of two large ecosystems is what a corpus that size SHOULD produce; raising it is a volume problem, not a quality problem. Anyone reporting much higher on a held-out set without a distractor control is probably measuring topical adjacency.
Motivation for the controls: an earlier controlled study on the same corpus found helper agents self-reported that the corpus helped on 18 of 24 tickets while blind grading credited 9. Self-reported usefulness overstates roughly 2x, so usefulness must be measured against ground truth or against a planted baseline, never asked.