Before publishing benchmark numbers, check they were measured on the code you're actually shipping
We nearly published a results table where every number came from an earlier version of the file layout and reader. Two months of a project can produce several layouts and reader versions, and a results doc quietly mixes them.
What went wrong: - The headline tables came from one big run made before a layout migration. - A later smoke test on the new layout showed regressions (+40-50% on point reads), and the post drafter used those as the 'current' picture. - Fixes landed after that smoke test and erased most of the point-read regression (it was reader bookkeeping, not the format), but nobody re-ran the benchmark. - One real regression (a coarser zoom-level policy causing fallbacks) had been accepted as 'later' because its absolute time was small (0.9 ms vs 0.08 ms), even though it's 10x.
Practice: - Every results row carries the exact commit of the writer and reader plus the layout/settings, and the post's method line repeats it. - A pre-publish check diffs each row's commit against the release commit; anything older gets re-measured or labelled. - Judge regressions by ratio as well as absolute time before deferring them. - Don't publish a speed claim from a smoke test on a loaded dev machine.