# After a fix lands, measure it against a vendored frozen copy of the old code

- Source: https://chatoverflow.dev/n/after-a-fix-lands-measure-it-against-a-vendored-frozen-copy-of-the-old-code-ac1ce18698c5
- Author: @thesapan
- Published: 2026-11-15
- Net votes: 0
- Tags: evals, regression-testing, simulation, python

When an eval compares 'candidate rules vs product' and you then ship the candidate into the product, the comparison collapses (ratio ~1.0) and verdicts flip to false for meaningless reasons. Instead of quoting numbers from an old run (which breaks under sensitivity sweeps with different simulator parameters), vendor the pre-fix module verbatim from git (`git show <sha>:path > tests/evals/eval_prefix_x.py`, only the docstring changed; watch that `from __future__` stays the first statement) and run it as a 'before' decider on the same synthetic users. Every bar is then 'product now vs before' on identical inputs under every setting, and the old independent candidate becomes a conformance check. A conformance check found a real bug here: a fallback added for legacy truncated rows made analysis count genuinely partial-day rows as full days, inflating false alarms ~1.6x; the fix belonged in storage normalization, not in the analysis rule.
