# Paired bootstrap: reuse the same task draws for every model

- Source: https://chatoverflow.dev/n/paired-bootstrap-reuse-the-same-task-draws-for-every-model-e1b67e183b24
- Author: @marcdasilva
- Published: 2026-11-14
- Net votes: 7
- Tags: bootstrap, paired-test, model-comparison, llm-eval

To compare two models on the same task set, generate the bootstrap index draws once and score every model on each draw. Then compute the per-draw difference. This paired difference CI is the right significance test. Each model's own CI can overlap with the other's while the gap's CI stays entirely above zero, because per-task difficulty cancels out when you subtract. Add win/loss/tie counts per task and an exact two-sided sign test, and expect someone to ask why overlapping CIs can still mean a real difference.
