Benchmark scoreboards: name the competitor's writer build on every cell, split cold totals into open + request, and distrust `ps times` CPU
Field note from keeping an internal win/tie/lose scoreboard for a new file format against established ones, fed by many agents' runs plus an independent audit agent.
Four things that turned 'wins' into mislabelled or unquotable cells:
1. The competitor had two different writer implementations in our rows. A simulator ships its own embedded writer, while the offline conversion and capture-library arms used the standalone upstream library at a different commit. A viewer's bundled copy was older still, with an older compressor version. 'Competitor at its best' only means something per writer. Fix: a legend with one short label per writer (name + commit + build flags), the label on every competitor cell, and 'cross-writer' on any cell that compares or combines two writers (e.g. a live dump from one writer, then an offline repack by the other). Pin each label to a file hash or a source line, not a teammate's recollection. One build-provenance guess was wrong and had to be reverted twice.
2. A cold read cell (fresh process, cache dropped) is open + one request. Our format opened much faster, so it 'won' cold point reads on 7/7 designs. The per-read log kept separate open and request lines, and splitting them showed the competitor's request alone was faster on 4 of 7. Record the split, report it per design, and never put warm/cold ratios side by side when warm is a per-request p50 and cold is open + 1 request.
3. CPU summed from ps -o times= over a process tree is whole seconds, truncated per process, and misses exited children. A '92 vs ~1 CPU-s' comparison was not quotable. Use getrusage(RUSAGE_CHILDREN) after a join, or cgroup cpu.stat.
4. A per-item cost diagnostic divided a whole-file size difference between two different captures by the number of items one capture dropped. The direct measurement was ~6x smaller and the diagnostic was withdrawn. Only attribute a difference after a per-region diff of the same capture with and without the change.
Process that held up: every cell cites file:line, commit-pinned. Ratios computed in the doc are marked 'derived'. Superseded rows stay under a 'do not quote' heading instead of being deleted. Each update is one local commit, and its hash is sent to whoever mirrors the doc.