Benchmark error bars: cluster single scores, pair comparisons
On SWE-bench Verified (500 tasks, 12 repos, Django = 231), the repo-clustered SE (CR1) of one model's solve rate was about 1.5x the iid binomial SE (2.7 vs 1.8 pp at 79%). For the paired difference between two models, the clustered and iid SEs were nearly equal (median 1.88 vs 1.93 pp), because repo difficulty cancels. So cluster per-model CIs, but compare models with a paired test (exact McNemar on discordant tasks). With about 12% task discordance, a McNemar test on 500 items has 80% power only at a gap of about 4.4 pp; a 1 pp gap gives about 8% power. I checked this by multinomial simulation.
Reproduced the power numbers on my own: 500 items, 12% discordant, exact two-sided McNemar (binomial on the discordant pairs, alpha 0.05), 4000 replicates per gap. Power was 7.9% at a 1 pp gap, 45% at 3 pp, 71% at 4 pp, 79.8% at 4.4 pp and 89% at 5 pp. The iid SE at 79% is sqrt(.79*.21/500) = 1.82 pp, which also matches. Rule of thumb from the same run: with 12% discordance, ranking two models from a single run on 500 items needs a gap of about 4-5 pp. Smaller gaps need more items or repeated runs per item.