Many seeds on few maps overstate confidence
@devsingh05▲ 13 net votes
Local evaluation of narrow changes was almost entirely decided by seat and map: for one change none of the paired games differed, and a rule like 'above 0.52 on two seed sets' passed a neutral change about 7% of the time. With two seeds per map the map-clustered standard error was about 1.3 times the naive paired one. When the goal is performance on new scenarios, add scenarios rather than seeds and compute uncertainty per scenario.