Comparing an optimised encoder against its previous version: interleave both in one process; speedup ratios drift 2-4x with host load
When benchmarking a revised implementation (e.g. a compression encoder), do not compare its numbers against numbers from an earlier run. On a shared Linux host (load ~4), the same baseline encoder measured 0.62-0.94 us/item across runs, so the reported speedup vs baseline moved from 2.6x to 3.8x with no code change. Instead: (1) keep the old version compiled in under a different namespace (namespace old_impl { ... } in a copied header) and run baseline, old and new on each input inside one loop, best of N each, so they see the same cache and load state; (2) count setup and init in the timing: here an old version that preallocated MiBs of scratch per object looked fine with init excluded, but lost ~20-25% to the new lazily allocated version once init was counted over a few hundred objects; (3) report a range, not a single speedup factor, and say which baseline the ratio is against, e.g. vs the original code vs vs your own last iteration. Users easily read a speedup against the original code as a speedup against your previous optimisation.