Put the confidence interval on the paired difference and cluster it
When comparing two models scored on the same items, do not report a separate CI per model and check whether they overlap — the errors are correlated, so that is the wrong test and it is far too conservative. Compute the per-item difference (a_correct minus b_correct) and bootstrap that. Then cluster the resample: if several rows share an underlying unit (same prompt, same item, same repeated-measure source), resample cluster ids with replacement and take every row of each drawn cluster, because rows within a cluster are not independent. This is not cosmetic — in one benchmark the naive row bootstrap gave [+0.0000, +0.1859] and just barely cleared zero while clustering by prompt gave [-0.0064, +0.1974] and contained it, flipping the headline from significant to unresolved on a 9.6 point gap. Also stratify and sweep the cut point: an ordering that reverses inside a stratum is the actual result, and sweeping the threshold across a range proves the reversal is not an artifact of where you cut.
Confirmed by simulation. Setup: 200 items x 5 repeats, logit item difficulty SD 1.5, item-level model interaction SD 1, 400 boards, 500 bootstrap draws. Under the null (no true difference): row bootstrap of the paired difference rejects 8.3% at nominal 5% (SE about 1.4 pp), the cluster-by-item bootstrap of the paired difference 4.5%, and the per-model CI overlap test 0.0%. At a true gap of 0.3 logit, power is 62% for the clustered paired CI and 3.8% for the overlap test. So the overlap test is not just conservative, it has almost no power here, and the row bootstrap is anti-conservative as soon as repeats share an item. One wording point: the clustered CI targets the population of items. If you want 'is A better on these exact items', the row-level CI with item fixed is the matching one, and it should be stated that way.