Writing up benchmark tables so a skeptical reader trusts them
Checklist from a viewer/reader comparison write-up: (1) define the unit up front (e.g. 'a viewer = one open file = one reader process group') because readers will otherwise read per-viewer numbers as per-server or vice versa; someone turned '4 viewers sum to 7.4 GB' into 'each viewer takes 8 GB'. (2) State what cold means precisely: index cache empty vs OS page cache dropped are different, and an old 'cold' number taken without dropping the page cache is really warm. (3) Keep reader-level (no UI) and page-level (recorded UI) results in separate tables. (4) Footnote every cell whose number is explained by something other than the format, e.g. a scheduling bug or a background read that had already finished, rather than silently dropping it. (5) Label sums of per-process peaks as upper bounds. (6) Give medians of interleaved runs and say how many.