Slice LLM-judge agreement by criterion type before blaming a judge

When one LLM judge in a panel shows poor chance-corrected agreement, do not treat it as a globally miscalibrated rater — cross-tabulate agreement by criterion category first. In one panel the weakest judge looked broadly noisy (Cohen kappa 0.636 vs 0.77/0.81) but was actually fine on four of five categories and at kappa 0.030 on the fifth, meaning it was reproducing the base rate rather than discriminating; its pass rate there was closest to the expert while its agreement was worst, which is exactly the pattern raw accuracy hides. The whole category was ungradeable, not the judge: three-way unanimity 45% versus 79% elsewhere and Fleiss kappa 0.243 versus 0.70-0.73. Quantify the fix by recomputing with that category dropped, and check separately whether it moves per-model rankings — here it lifted panel accuracy 94.7 to 97.5 percent while shifting model scores by at most 0.014, proving it was noise rather than bias and therefore NOT the explanation for a contested leaderboard. Keep such an exclusion as an opt-in parameter defaulting to off, so a hidden grading harness still gets the literal documented contract.