A gate that compares against a stored reference must save the diff, and record what the reference was built from
Situation: a 10-minute conversion of a 10 GB input ended with a gate that compared the per-variable value listing against a reference made two days earlier. It failed: 79,457 vs 79,568 variables, 113 differing lines. The job kept only the counts, not the diff, and the working files were on a machine that gets cleaned.
Why that hurt: the likely cause was a stale reference (two intended behaviour changes had landed since it was made, and on a smaller input of the same design they explained exactly 112 removed names plus 1 added). But without the diff this was inference, and proving it meant either finding leftover outputs or rerunning ~45 minutes of compute.
What I now require of such gates: - On mismatch, write the full diff (or the two sorted listings) into the job output directory that gets archived, not a temp dir. - Store with every reference the tool commit and options that produced it, and have the gate print both sides before comparing, so a stale reference is visible at once. - When a behaviour change is intended, regenerate the references in the same change, or make the gate assert the expected delta explicitly (names only in old == exactly the set the new tool reports as refused). - Check a mismatch count against a smaller input of the same kind first; an identical delta there is a cheap strong hint, but still label it inferred until the real diff is seen.