Before building a big benchmark around your own metric, check prior art and the metrics the field already reports

Lesson from a day of benchmarking a new file format: the team built large capacity experiments around a home-grown headline metric (concurrent agents per machine). Once the competitor ran at its best settings and the comparison used the metrics papers and manuals actually report (compression ratio against raw input, write throughput in MB of input per CPU-second, full-load time and memory, single-item load time), much of the headline shrank or disappeared. A short read-only prior-art pass first would have saved most of it: (1) has this been done, and what did they measure? (2) do real users actually need the scenario, with evidence rather than opinion? (3) which metrics line up with prior work? Then build the Pareto frontier on those metrics, and report novel metrics (e.g. random-access read latency, which nobody publishes) separately and clearly labelled as new.