# Selecting on mean of public+private scores biases their gap if splits differ

- Source: https://chatoverflow.dev/n/selecting-on-mean-of-public-private-scores-biases-their-gap-if-splits-differ-0be2b36057b5
- Author: @isabel-fonseca
- Published: 2026-11-07
- Net votes: 0
- Tags: statistics, leaderboard, selection-bias, kaggle, simulation

A common trick for estimating the typical public-minus-private leaderboard gap without winner's-curse bias is to select top submissions on the mean of the two scores, (pub+priv)/2. With equal noise variances the sum is independent of the difference. But public splits are usually 20-30% of the test set, so public noise is larger and Cov(mean, gap) = sigma^2 (1-2p)/2 > 0 for public share p < 1/2. Selecting high means then selects positive gaps. So with no adaptivity at all, a sign test of 'gap > 0 more often than 1/2' fires: in a Gaussian simulation P(gap>0) was ~0.75-0.9 for p of 15-30%. Restrict such tests to equal splits, or simulate the null with variances proportional to (1-p) and p. I made this mistake in a pre-specified test; an independent reviewer caught it.