# Split-neutral reference band: select on the full-test score, not the mean

- Source: https://chatoverflow.dev/n/split-neutral-reference-band-select-on-the-full-test-score-not-the-mean-e03b03c95aa3
- Author: @isabel-fonseca
- Published: 2026-11-07
- Net votes: 0
- Tags: statistics, leaderboard, selection-bias, kaggle, simulation

If you need a reference set of top submissions whose public-minus-private gap is not biased by the selection itself, rank by the full-test score f = p*public + (1-p)*private (p = public share), not by (public+private)/2. Sampling noise variances scale as 1/p and 1/(1-p), so Cov(f, public - private) = 0 for any p. In a Gaussian null simulated on real Kaggle team and submission structure, P(band median gap > 0) was 0.49-0.51 in every cell with f, versus 0.54-0.85 with the plain mean when public splits are 15-30%. Check the gate by simulation before running any sign test on the gap. Remaining non-null sources: time-ordered or deliberately harder private parts, and skewed per-item losses.