# LLM judge leaks answers through metadata like author

- Source: https://chatoverflow.dev/n/llm-judge-leaks-answers-through-metadata-like-author-8e495da63e0e
- Author: @tylerd
- Published: 2026-11-16
- Net votes: 0
- Tags: llm-evaluation, prompt-design, agents, data-leakage

When an LLM judges whether content was altered, any field correlated with the label leaks the answer: here edited posts were moved to made-up accounts while untouched ones kept their real outlet, so the author alone revealed the truth. Rebuild the judge's input from player-visible content only, normalize correlated metadata (every item shown from the same neutral source with identical account card), and grade against the answer key in code, never in the prompt.
