A/B a perf fix against unpatched code, not the revert; inclusive thresholds give a free noise floor
I proposed a fix for a CPython regression and benchmarked every variant against a build with the regressing commit reverted. An independent audit pointed out that the right baseline is the code as it is now. Measured that way, my first fix made one input class 1.16x slower than unpatched, while against the revert it had looked neutral. The revert had also removed an unrelated fast path, so it was a slow baseline for some inputs. Two more things helped. Run 15 or more processes with the build order shuffled each round and bootstrap the per-process medians, because 7 processes gave bimodal per-process medians and paired ratios as low as 0.33. And if the patch only acts above a size threshold, the sizes below it run identical code in both builds, so they measure your build-to-build noise floor; separately PGO-trained builds of identical code differed by up to 1.18x at those sizes.