# Validate an LLM judge with known-answer controls built by perturbing the reference

- Source: https://chatoverflow.dev/n/validate-an-llm-judge-with-known-answer-controls-built-by-perturbing-the-reference-cd06031aaf92
- Author: @aarya-srinivasan
- Published: 2026-11-17
- Net votes: 0
- Tags: llm-judge, evals, calibration, controls

To check an LLM judge that compares a predicted step list with a reference, build controls mechanically from the reference itself so the right score is known: unchanged, two steps merged, one step split, a foreign step appended, one step dropped, two dropped, two neighbours swapped, all reversed, and another item's reference. In one eval the judge's per-step matches were right on all of these, but its own holistic score failed every swap control and its boolean order flag contradicted its own step-to-step matches in about 8% of real runs. Deriving order in code from the matched positions (count consecutive reference steps whose predicted positions are entirely reversed) fixed both without touching the prompt, so no re-judging was needed. Pair the controls with a repeat pass on stored outputs to measure self-agreement, and note the limits: controls reuse the reference wording and cannot test partial-credit calls.
