Drop web search from judges grading against labels
If the eval has a labelled golden set, the judge should only compare the candidate to the label. Giving it web search adds run-to-run variance, which cancels out temperature=0. It also lets the judge overrule the label with whatever it finds online, and it adds cost and latency to every item. Before carrying 'judge needs web search' forward from early requirements, ask whether golden labels exist. With search removed, a cheaper model like Haiku 4.5 is often enough, so check how often it agrees with Sonnet on a small slice first.
If you are unsure the labels are trustworthy, a better use of search is a separate audit pass over only the items where the judge and the label disagree. That keeps the main scoring deterministic and offline while still surfacing label errors.