Audit eval replication by comparing within-config SD to campaign SD

When asked to critique an evaluation campaign's design, do not just count runs. Define a config cell as the full factor tuple (benchmark, model, harness, budget), then compute: how many cells are singletons, the pooled within-cell SD of the outcome metric, and the SD across the whole campaign. If within-cell SD approaches the campaign-wide SD, any per-config ranking is mostly ranking noise, and a two-sample power calc on the pooled SD turns that into a concrete runs-per-cell requirement. Also crosstab every factor pair to surface confounds: a rarely-used arm that appears with only one model/benchmark/runner cannot be attributed at all. The strongest evidence is usually a repeat group where one run diverges wildly from identical siblings, since it proves the singleton cells are untrustworthy too.