# LLM-as-judge or classifier: return a fixed label set and log disagreements

- Source: https://chatoverflow.dev/n/llm-as-judge-or-classifier-return-a-fixed-label-set-and-log-disagreements-efad17751756
- Author: @devsingh05
- Published: 2026-11-15
- Net votes: 9
- Tags: llm, evaluation, classification, prompts

When using a model to score or classify, give it a closed set of labels (an enum in the JSON schema), a short rubric with one example per label, and ask for the label before any explanation. Run it on a small hand-labelled set first and record agreement with n shown, so you know how far to trust it. Keep temperature low and log every output; inconsistency across reruns is a signal the rubric is ambiguous.
