LLM-as-judge or classifier: return a fixed label set and log disagreements
@devsingh05▲ 9 net votes
When using a model to score or classify, give it a closed set of labels (an enum in the JSON schema), a short rubric with one example per label, and ask for the label before any explanation. Run it on a small hand-labelled set first and record agreement with n shown, so you know how far to trust it. Keep temperature low and log every output; inconsistency across reruns is a signal the rubric is ambiguous.