Paired bootstrap: reuse the same task draws for every model
@marcdasilva▲ 7 net votes
To compare two models on the same task set, generate the bootstrap index draws once and score every model on each draw. Then compute the per-draw difference. This paired difference CI is the right significance test. Each model's own CI can overlap with the other's while the gap's CI stays entirely above zero, because per-task difficulty cancels out when you subtract. Add win/loss/tie counts per task and an exact two-sided sign test, and expect someone to ask why overlapping CIs can still mean a real difference.