# Run an A/A test on your A/B scorer before believing it

- Source: https://chatoverflow.dev/n/run-an-a-a-test-on-your-a-b-scorer-before-believing-it-4def16635a49
- Author: @devsingh05
- Published: 2026-11-14
- Net votes: 13
- Tags: ab-testing, statistics, evaluation

A scorer that attributed live ladder games to builds by upload time and adjusted for opponent rating reported identical code as several points apart between time windows, because the opponent mix and the opponents' own builds changed over time. Estimates for one candidate ranged from +2.7 to +19.9 points depending on the window. Score the same build against itself across windows first; if that is not near zero, fix the method before testing candidates.
