Sweep the unit-of-observation axis before trusting a bootstrap CI
When auditing a pairwise-preference eval where some items carry repeated/duplicate labels, the choice of analysis unit silently drives the result. Three defensible units gave: row-level (double-weights repeated items) diff -0.172 CI [-0.290,-0.063] excludes zero; keep-first-label-per-item -0.133 CI [-0.289,+0.022] straddles zero; majority-vote-per-item -0.237 excludes zero. Majority-vote is the trap: it drops items with no majority, i.e. exactly the ones raters disagreed on, so it selects on the outcome and inflates the winner. Also check this axis before concluding a published headline number is irreproducible — a 144-variant robustness grid missed the true config because its pair axis only implemented majority-vote and never keep-first. Rule: when a headline claim rests on a judgment call you flagged as uncertain, test the alternative most likely to break it, not the nearest one.
The majority-vote point can be made exact. Take an item with two labels, each preferring A with probability p. Majority vote with ties dropped keeps the item with probability p^2+(1-p)^2 and then says A with probability p^2/(p^2+(1-p)^2). The difference is therefore roughly doubled for small effects: p=0.55 gives keep-first 0.10 vs majority 0.198 (about half the items dropped); p=0.6 gives 0.20 vs 0.385; p=0.7 gives 0.40 vs 0.69. This is a change of estimand (the sign of the item-level preference) as much as a selection bias, so a majority-vote number should never be compared with a per-label number. Keep-first, or averaging the labels within each item, keeps the per-label estimand.