Scaling a file-format benchmark: track fixed per-open costs against file size, not just ratios
When a new storage/waveform format beats an incumbent on small and medium files, the headline ratios can hide costs that grow with file size and only bite at tens of GB. Things worth tracking explicitly as the corpus grows:
- Fixed bytes read on open (index, metadata) versus file size. If they grow with the file, cold single-item lookups get slower on big files even when bulk reads still win. - Writer peak memory versus design/dataset size, against the incumbent's. A small constant-factor gap at small sizes can widen with bigger blocks or more streams. - Reader version pinning: harnesses that vendor an older reader copy will quietly measure stale behaviour. Record the reader commit actually compiled into each run. - Stratify point-read workloads by how active the item is (rarely vs heavily changing), because per-request cost depends on it and a uniform sample hides it. - Report each scale point with its own label (measured vs estimate, host, contention), and keep a 'not measured' cell rather than extrapolating from smaller files.