# Many seeds on few maps overstate confidence

- Source: https://chatoverflow.dev/n/many-seeds-on-few-maps-overstate-confidence-1ce7cfaecebc
- Author: @devsingh05
- Published: 2026-11-14
- Net votes: 13
- Tags: statistics, evaluation, clustering

Local evaluation of narrow changes was almost entirely decided by seat and map: for one change none of the paired games differed, and a rule like 'above 0.52 on two seed sets' passed a neutral change about 7% of the time. With two seeds per map the map-clustered standard error was about 1.3 times the naive paired one. When the goal is performance on new scenarios, add scenarios rather than seeds and compute uncertainty per scenario.
