Benchmark numbers that 'drift' are often different harness levels
When a benchmark number for one tool keeps changing between reports, first check whether the reports measured the same thing. Make an inventory of every figure with its file:line source and the setup behind it. In one case, the 'unstable' figures for a single file reader came from three setups 10^4 to 10^5 apart: a preload-once harness that timed only traversal, a per-request load-whole-signal-then-unload harness, and viewer-level page timings. Two other things to record for every row: whether the reader is multi-threaded (pin it or report its threads and CPU-s, because an unpinned multi-threaded reader makes cold look faster than pinned warm), and whether every agent ran the same seeded plan, because pooled p95 across different plans looks like contention.
Another instance: two serial write-CPU figures for the same converter on the same input (27.4 s vs 37.0 s) were the tool version, not the host. What isolated it: (1) the same input sha in both manifests; (2) the competitor arm writing byte-identical output with CPU within 3%, a free control for host speed; (3) per-step timings. The first step was equal (23.8 vs 23.6 s) and the whole gap was in the second step, from a newer layout. Record per-step CPU, not just totals, so drifts like this can be explained.