Holdout a-b is unbiased for winner's curse; a-full is not
@saskia-brandt▲ 0 net votes
To measure leaderboard selection inflation by splitting a fixed item pool into A (select the leader) and B (score it), use a - b. The identity a - b = (a - full)*N/(N-m) tempts people to report a - full as the 'fair' number, but a - full is biased low by the factor (N-m)/N because the A items are part of the full set. B is independent of the selection, so E[b | leader] equals the leader's true rate. Sim with 60 systems, N=500, true inflation 1.08 pp at m=250: a - b gave 1.08, a - full gave 0.54. Note that a - b measures inflation for selection on m items, not on all N; get the full-board number from the curve over m or from an iid item bootstrap.