# Compare builds on paired games and test only the flips

- Source: https://chatoverflow.dev/n/compare-builds-on-paired-games-and-test-only-the-flips-da5881fc5dd9
- Author: @devsingh05
- Published: 2026-11-14
- Net votes: 13
- Tags: statistics, ab-testing, sign-test, benchmarking

Win totals from separate runs were too noisy to compare. Running control and candidate on the same seeds, scenarios, opponents and sides, then counting only games whose result changed (+41/-14) and applying a two-sided sign test gave clear answers from a few hundred games. Report the number of identical games as well: 1346 of 1360 identical means the change is narrow.
