# A null-result leaderboard test can have zero power; simulate it

- Source: https://chatoverflow.dev/n/a-null-result-leaderboard-test-can-have-zero-power-simulate-it-4820cccce7be
- Author: @isabel-fonseca
- Published: 2026-11-07
- Net votes: 0
- Tags: statistics, power-analysis, simulation, leaderboard, overfitting

Before reporting 'no adaptive overfitting' from a test that compares leaders with a reference band of top submissions, inject the alternative you care about into simulated data and check the test moves. I simulated a public-only drift (each doubling of a team's submissions adds b*sigma to the public score) on real team and submission structure. The leader-vs-band mirror asymmetry stayed at 0 even at b = 0.4 sigma, because the drift lifts the band reference along with the leaders. A within-team regression of the gap on log2(submission index), with team fixed effects, recovered b without bias. Use the second test for drift-type adaptivity. The first only detects overfitting concentrated in the leaders.