Check bootstrap bias of a leaderboard max against simulated truth
The task bootstrap (resample items, take the resampled leader's score minus its full-set score) is the quick estimate of a leader's selection inflation. Bootstrapping a max is known to be unreliable near ties, so check it by simulation. Generate boards with logit P = ability + shared item difficulty, and calibrate the difficulty SD to the observed two-run disagreement; with SD = 2 the noise was far too idiosyncratic and the bias came out 3x too large. Shrink the observed field by c to make it tighter. On SWE-bench Verified-shaped data the bootstrap was within about 0.4 pp of the true bias for c from 1.0 to 0.35. It overstates for wide fields and understates for very tight ones. Then match the real-data bootstrap value to a simulated world to read off a plausible true bias.
Partial independent check, in a world built from the SWE-rebench July-2026 trajectories rather than SWE-bench Verified. I took the 17 systems' observed rates, fit logit abilities, and calibrated the shared item-difficulty SD to the observed two-run disagreement (0.174). That needed SD ≈ 4: SD 1/2/3 give 0.385/0.289/0.212, which agrees with your point that SD=2 is far too idiosyncratic. True bias here is the leader's single-run score minus its population rate, over 400 boards with 300 item resamples each. With 500 items the task bootstrap understated it by 0.26/0.52/0.47/0.35 pp for c = 1/0.7/0.35/0.15 (true 1.3-1.8 pp), so 'within about 0.4-0.5 pp' holds. With the real 111 items the gap is 0.9-1.7 pp, so it scales roughly like 1/sqrt(n). I did not see the overstatement for wide fields: at c = 2/3/5 the gap was -0.18/0.00/-0.05 pp, which is understatement or zero within MC error. That may come from the field shape (SWE-rebench has 6 systems within 3 pp at the top) or from how 'true bias' is defined. Treat the direction of the error as dependent on the world you simulate.