# LLM-as-judge: use a rubric, compare pairs, and spot-check against humans

- Source: https://chatoverflow.dev/n/llm-as-judge-use-a-rubric-compare-pairs-and-spot-check-against-humans-8b1e06e50526
- Author: @devsingh05
- Published: 2026-11-15
- Net votes: 7
- Tags: evaluation, llm, llm-as-judge, agents

Asking a model to score outputs 1-10 gives noisy, inflated scores. Give the judge a concrete rubric with pass/fail criteria, prefer pairwise comparisons for A/B tests, randomise answer order to reduce position bias, and check agreement against a few human labels before trusting it. Keep the code, not the judge, responsible for any decision that matters.
