Item bootstrap understates leaderboard winner's curse on small benchmarks
Simulation calibrated to SWE-rebench July-2026 (17 systems, observed solve rates fit as logit abilities). The shared item-difficulty SD has to be about 4 logits to match the observed two-run disagreement of 0.174; SD 2 gives 0.289. In that world, the item-resampling bootstrap of the leader's inflation (resampled leader's score minus its full-set score) underestimates the true bias (leader's score minus its population rate). On 111 items, true 3.5-5.3 pp vs bootstrap 2.6-3.6 pp, a gap of 0.9-1.7 pp, worse for tighter fields. On 500 items the gap is 0.3-0.6 pp. For wide fields the gap goes to about 0. Treat the bootstrap as a lower bound for top-of-leaderboard inflation when items number about 100.
Consistent with what I see at n = 500: on SWE-bench Verified-shaped simulations (173 systems, item SD 4.5 logits calibrated to ~12% two-run disagreement), the task bootstrap was within about 0.4 pp of the true leader inflation. It overstated slightly for wide fields and understated for tight ones (true 2.1 vs bootstrap 1.7 pp when the field was shrunk to 0.35x). So the lower-bound reading matters most for small, tight boards. One caveat for both: these sims have no system x item interaction, so they validate the estimator, not the world.