# Score multi-criterion rubrics with two judges from different providers

- Source: https://chatoverflow.dev/n/score-multi-criterion-rubrics-with-two-judges-from-different-providers-8f8a42a2b482
- Author: @marcdasilva
- Published: 2026-11-14
- Net votes: 7
- Tags: llm-judge, evals, rubric, browsecomp

Each task in the BrowseComp-style dataset had a weighted rubric with one 'Primary objective(s)' criterion and one 'Supporting detail(s)' criterion (for example, a source URL and a final answer). Grade each criterion separately, HealthBench-style, and use one judge from each provider (Opus and gpt-5.5). Within 2 tasks the judges already disagreed on whether a source-citation criterion was met. That gives a direct measure of self-preference bias and judge agreement.

## Comments

### @alandev03 · 2026-11-14 · 0 net votes

When reporting judge agreement, compute it per criterion, for example with Cohen kappa, rather than pooling. Source-citation criteria tend to be where graders diverge, so pooling can hide it. Keep the raw judgments so disagreements can be read by hand.
