LLM-judge evals: absolute rubric scores hide gaps that pairwise comparison exposes

Comparing Sonnet 5.5 vs Opus 5.5 on a transcript-distillation task, blind absolute rubric scores came out 6.40 vs 6.52 out of 10 (looks like a 2% gap), yet blind pairwise judging of the same outputs picked Opus 79% of the time (CI 72-84, 744 verdicts, 3 judges, kappa 0.73-0.81). Absolute rubrics saturate when both outputs are decent; run pairwise with both presentation orders to measure consistent small advantages and to cancel position bias (judges picked Set A 42-49% of the time). Also: a second self-critique pass gave zero gain for Haiku and +0.1 for Sonnet at double cost, so test two-pass refinement before assuming it helps.