Scaling a benchmark suite from small files to 10-50 GB: what changes in the plan

Lessons from moving a file-format benchmark (conversion, write-time overhead, serving memory, reads) from sub-GB inputs to 10/25/50 GB tiers.

- Define tiers by one named size (e.g. the competitor's native dump size), and keep the same design across tiers by extending run length. That gives size-vs-length curves instead of unrelated points. - Per-tier sums before any launch: input + every intermediate + outputs + copies on local NVMe, and time per step taken from past logs at the nearest size. Steps that take minutes at 1 GB (format conversion, per-item content hashing) take about an hour at 10 GB+. - At this size, full per-item content checks can dominate the run. Use full totals (e.g. total event counts) plus a seeded per-item sample, and label it as such. - Fan out: one VM per (design, tier, track), each self-terminating behind a 'results pulled' handshake plus a hard cap. Keep one independent auditor reviewing every plan and result, so parallel tracks don't drift on settings or labels. - Fix known read-path issues before the big read runs, or label which reader version each cell used. Small-file benchmarks hide fixed open and index costs that dominate, or vanish, at scale. - Shared lab hosts usually lack the disk for 25-50 GB tiers. Decide early which tool chains need dedicated machines (and any tool-install policy questions) instead of discovering it mid-run.