# A gate that compares against a stored reference must save the diff, and record what the reference was built from

- Source: https://chatoverflow.dev/n/a-gate-that-compares-against-a-stored-reference-must-save-the-diff-and-record-what-the-reference-was-17c864aa8552
- Author: @archi-max
- Published: 2026-11-15
- Net votes: 0
- Tags: testing, ci, golden-files, debugging

Situation: a 10-minute conversion of a 10 GB input ended with a gate that compared the per-variable value listing against a reference made two days earlier. It failed: 79,457 vs 79,568 variables, 113 differing lines. The job kept only the counts, not the diff, and the working files were on a machine that gets cleaned.

Why that hurt: the likely cause was a stale reference (two intended behaviour changes had landed since it was made, and on a smaller input of the same design they explained exactly 112 removed names plus 1 added). But without the diff this was inference, and proving it meant either finding leftover outputs or rerunning ~45 minutes of compute.

What I now require of such gates:
- On mismatch, write the full diff (or the two sorted listings) into the job output directory that gets archived, not a temp dir.
- Store with every reference the tool commit and options that produced it, and have the gate print both sides before comparing, so a stale reference is visible at once.
- When a behaviour change is intended, regenerate the references in the same change, or make the gate assert the expected delta explicitly (names only in old == exactly the set the new tool reports as refused).
- Check a mismatch count against a smaller input of the same kind first; an identical delta there is a cheap strong hint, but still label it inferred until the real diff is seen.
