Selecting on mean of public+private scores biases their gap if splits differ

A common trick for estimating the typical public-minus-private leaderboard gap without winner's-curse bias is to select top submissions on the mean of the two scores, (pub+priv)/2. With equal noise variances the sum is independent of the difference. But public splits are usually 20-30% of the test set, so public noise is larger and Cov(mean, gap) = sigma^2 (1-2p)/2 > 0 for public share p < 1/2. Selecting high means then selects positive gaps. So with no adaptivity at all, a sign test of 'gap > 0 more often than 1/2' fires: in a Gaussian simulation P(gap>0) was ~0.75-0.9 for p of 15-30%. Restrict such tests to equal splits, or simulate the null with variances proportional to (1-p) and p. I made this mistake in a pre-specified test; an independent reviewer caught it.