A 160x speedup across hosts and inputs shrank to 5x on the same host: rerun before you quote
We compared a new failure-localisation method (find where a buggy simulation first diverges from a good one) against the existing one. The first comparison used numbers from two different setups: the new method on short ~6k-cycle bench runs on one host, the old one on a long testbench run on another. That gave ~160x fewer CPU-seconds per failure.
Rerun on the same host, the same 4 failures and the same seeds, it was 0.90 vs 4.6 CPU-s (about 5x). A third, cheaper variant of the old method came in at 1.16 CPU-s, close to the new one. Most of the 160x came from the input size, not the method. The real advantage turned out to be qualitative: only the new method gave the exact first-difference cycle.
Practice: - Label any cross-host or cross-input ratio as an estimate until it's rerun. - Include the best-configured version of the baseline, not just its default. - Report the per-run overhead (here, hashing cost on passing runs) next to the per-failure saving, and give the break-even failure rate. - Run the comparison under a hard memory cap. The first attempt at the old method parsed whole-run dumps and hit our cap, which killed it cleanly instead of starving the shared host.
Follow-up from the review: even the same-host 5x needed one more correction. The new method's per-failure figure left out its reference run and the overhead every passing run pays to make it possible. The baseline variants left out triage. Counted like for like, the new method was only ~7% cheaper than the best baseline variant, under our bar. Because its cost scales with the number of regression runs, it breaks even at ~10 runs per failure and costs ~2x at 100. Add an 'included' column that lists exactly what each row counts, and spread fixed per-run costs over the runs-per-failure ratio of the real workload.