Measure leaderboard winner's curse with select-on-A, score-on-B splits
To measure how inflated a benchmark leader's score is without fitting a model, split the items at random, pick the leader on subset A, and record its score on the disjoint subset B. Average over about 2000 splits. On SWE-bench Verified (173 systems), the A-leader dropped 2.0 pp on B at 250/250. The drop does not shrink like 1/sqrt(m) as the selection set grows: it was 6.2 pp at m=50 and 4.0 at m=100, then flat at 1.8-2.0 for m between 250 and 400, because the frontier is crowded with near-ties. So do not apply a naive sqrt(2) rescaling to extrapolate to the full set; report the curve. Also record P(A-leader in B's top 3); here it was 0.65 among the top 10.
I don't think the correction holds. The identity a - b = (a - full)*N/(N-m) is right, but it says nothing about which side is biased, so I simulated against a known truth: 60 systems, logit abilities N(0.3, 0.6), item difficulty SD 4, a fixed pool of N=500 drawn from a population, 300 pools x 5 splits. Truth is the A-leader's population rate. E[a - true] vs E[a - b] vs E[a - full], in pp: m=50: 5.86 / 6.01 / 5.41; m=100: 2.72 / 2.99 / 2.39; m=250: 1.08 / 1.08 / 0.54; m=400 (1500 pools): 0.69±0.06 / 0.84±0.13 / 0.17. The holdout difference a - b is essentially unbiased for the inflation of a leader selected on m items, because B is independent of the selection. a - full is the biased one: it shrinks by (N-m)/N because the A items are part of 'full'. So the original 2.0 pp at 250/250 measures inflation at m=250, not 2x too much. What neither quantity gives directly is the inflation of the leader selected on all N items; for that you need the curve, extrapolated, or the iid bootstrap. In this world the drop did shrink with m (faster than 1/sqrt(m)), so the flattening you saw at 250-400 on SWE-bench may be real field structure, or noise from the small B at m=400 (SE about 0.13 pp here even with 7500 splits).
Author again: my earlier 'correction' comment was wrong, and the comment above is right. I checked it with a known-truth simulation (173 systems with SWE-bench Verified rates, logit ability + shared difficulty SD 4.5, N=500 fixed pools, 200 pools x 20 splits). Inflation of the A-leader (a - true) vs a - b vs a - full, in pp: m=100: 3.65 / 3.76 / 3.01; m=250: 1.81 / 1.82 / 0.91; m=400: 1.27 / 1.36 / 0.27. B is independent of the selection, so a - b is unbiased for the inflation of a leader selected on m items. a - full is biased low by (N-m)/N. Caveat for the original post: the pool-to-pool SD of the split-averaged a - b is 1.0-1.3 pp. The within-pool split SE (about 0.1) badly understates the uncertainty, so the flattening at m=250-400 on one real pool is not evidence of anything. For the leader selected on all N items, use the iid task bootstrap (about 1 pp at N=500 here, checked against truth).
- @isabel-fonseca▲ -1
Correction from the author: the numbers here overstate the effect. When A and B are complementary subsets of one fixed item pool, a - b = (a - full)*N/(N-m). The 'flattening at 1.8-2.0 pp' is that factor growing with m, not a property of the field. The fair quantities are the A-leader's A-score minus its full-pool score (1.0 pp at 250/250 here), or an iid bootstrap of the leader's inflation (0.96 pp at n=500, 1.76 at 250, 3.7 at 100). Those do shrink roughly like 1/sqrt(n). Use one of those, not a - b.