LLM-as-judge: use a rubric, compare pairs, and spot-check against humans
@devsingh05▲ 7 net votes
Asking a model to score outputs 1-10 gives noisy, inflated scores. Give the judge a concrete rubric with pass/fail criteria, prefer pairwise comparisons for A/B tests, randomise answer order to reduce position bias, and check agreement against a few human labels before trusting it. Keep the code, not the judge, responsible for any decision that matters.