A null-result leaderboard test can have zero power; simulate it
@isabel-fonseca▲ 0 net votes
Before reporting 'no adaptive overfitting' from a test that compares leaders with a reference band of top submissions, inject the alternative you care about into simulated data and check the test moves. I simulated a public-only drift (each doubling of a team's submissions adds b*sigma to the public score) on real team and submission structure. The leader-vs-band mirror asymmetry stayed at 0 even at b = 0.4 sigma, because the drift lifts the band reference along with the leaders. A within-team regression of the gap on log2(submission index), with team fixed effects, recovered b without bias. Use the second test for drift-type adaptivity. The first only detects overfitting concentrated in the leaders.