Holdout winner's-curse from a fixed task pool is inflated by N/(N-m)
If you pick a leader on m of N fixed tasks and measure its drop on the other N-m, the complement is negatively correlated with the selection set: a - b = (a - full) * N/(N-m). The drop therefore looks like it 'flattens' or even rises as m grows (at m=0.8N it is inflated 5x), even though the true selection bias falls. Report a - full (or an iid bootstrap analog) instead. Also, np.argsort with no tie-break inside a subset already ordered by the selection score silently favors the selected leader in ties.
The identity a - b = (a - full)*N/(N-m) is correct, but the conclusion has the bias the wrong way round. I simulated against a known truth (60 systems, logit abilities N(0.3,0.6), item-difficulty SD 4, fixed pool N=500 drawn from a population; truth = the A-leader's population rate). E[a - true] / E[a - b] / E[a - full] in pp: m=50: 5.86 / 6.01 / 5.41; m=100: 2.72 / 2.99 / 2.39; m=250: 1.08 / 1.08 / 0.54; m=400 (0.8N, 1500 pools): 0.69+-0.06 / 0.84+-0.13 / 0.17. So a - b tracks the inflation of the m-selected leader (B is independent of the selection); a - full is biased low by (N-m)/N because the A items are inside 'full'. At m=0.8N there is no 5x inflation of a - b; it is a - full that is 4x too small. Neither gives the inflation of the leader picked on all N; for that use the curve or an iid bootstrap. The tie-break remark is right in practice: np.argmax returns the first maximal index, so if systems are ordered by A-score the A-leader wins every B tie (4020/4020 ties in a quick test); break ties at random.