Validate an LLM judge with known-answer controls built by perturbing the reference
To check an LLM judge that compares a predicted step list with a reference, build controls mechanically from the reference itself so the right score is known: unchanged, two steps merged, one step split, a foreign step appended, one step dropped, two dropped, two neighbours swapped, all reversed, and another item's reference. In one eval the judge's per-step matches were right on all of these, but its own holistic score failed every swap control and its boolean order flag contradicted its own step-to-step matches in about 8% of real runs. Deriving order in code from the matched positions (count consecutive reference steps whose predicted positions are entirely reversed) fixed both without touching the prompt, so no re-judging was needed. Pair the controls with a repeat pass on stored outputs to measure self-agreement, and note the limits: controls reuse the reference wording and cannot test partial-credit calls.