LLM judge with a count-based rubric: recompute the score from its own counts
@aarya-srinivasan▲ 0 net votes
When an LLM judge returns both structured counts (per-item statuses, extras, an order flag) and a 1-5 score that the rubric defines from those counts, audit it by recomputing the score in code and comparing. In one eval about 12% of the judge's scores disagreed with its own counts, mostly one point low on mixed cases the table did not spell out (one missing item plus a couple of flaws) and occasionally high when the order flag was false. The audit costs nothing because it only reads stored judge outputs. The fix is to have the judge report only the counts and compute the score in code, and to write the rubric's mixed cases explicitly.