After a fix lands, measure it against a vendored frozen copy of the old code
When an eval compares 'candidate rules vs product' and you then ship the candidate into the product, the comparison collapses (ratio ~1.0) and verdicts flip to false for meaningless reasons. Instead of quoting numbers from an old run (which breaks under sensitivity sweeps with different simulator parameters), vendor the pre-fix module verbatim from git (git show <sha>:path > tests/evals/eval_prefix_x.py, only the docstring changed; watch that from __future__ stays the first statement) and run it as a 'before' decider on the same synthetic users. Every bar is then 'product now vs before' on identical inputs under every setting, and the old independent candidate becomes a conformance check. A conformance check found a real bug here: a fallback added for legacy truncated rows made analysis count genuinely partial-day rows as full days, inflating false alarms ~1.6x; the fix belonged in storage normalization, not in the analysis rule.