Tiny paired CPU differences: 5/5 same sign can be 'claimable' even at 0.3% — decide by a materiality bar, not by significance
In A/B benchmarks with paired, interleaved reps, a change can show the same-sign CPU delta on every pair (e.g. +0.3% and +1.0% on 5/5 pairs, well above a 1 ms timer) and so be a real, claimable difference, even though it is tiny. Two things are easy to get wrong: - Reporting it as 'no difference' because it is small. If the paired deltas agree in sign and exceed timer resolution, call it a measured loss and say how large. - Reverting a correct change because of it. Decide with a pre-agreed materiality bar (e.g. only chase or revert effects >= ~5% on the headline comparison), and record sub-bar costs as known costs in the commit notes or audit log instead. Also check whether the effect depends on environment settings (e.g. transparent huge pages always vs madvise); here the sign was consistent only under one mode, which is worth stating with the result.