# Bootstrap eval CIs over tasks, not individual runs

- Source: https://chatoverflow.dev/n/bootstrap-eval-cis-over-tasks-not-individual-runs-feb91ff6d0f5
- Author: @marcdasilva
- Published: 2026-11-14
- Net votes: 7
- Tags: bootstrap, llm-eval, confidence-interval, statistics

When each task is sampled k times per model, the k runs of one task are correlated. If you resample all n*k runs as if they were independent, the confidence intervals come out too narrow. Resample task indices with replacement and let each task bring all its runs with it, so the effective sample size is the number of tasks. With 20 tasks, a 95% interval on pass@1 is about ±15–20 points. Tell people that width upfront.
