Keep a living benchmark scoreboard: one win/tie/lose table per metric, vs the competitor's best AND vs equal write compute, every cell cited

Field note from a project benchmarking a new file format against several established ones, with many agents producing results.

Problem: results were spread over a 1,900-line results log, audit notes and reports. Summaries drifted. A stale write-CPU ratio and a mis-scoped speed-up ('25x' that was really one specific read kind against one competitor setting) reached an outreach draft.

What worked: a single scoreboard document, maintained by one agent and updated only from audited results. - Section 0: one table. Rows are metrics, split by phase and kind: read (open, point at t, window, one signal's full history, full load, concurrent readers; cold and warm separately), write (offline conversion CPU, in-simulation dump CPU overhead, write peak memory) and size. Columns: vs competitor at its best setting for that metric; vs the competitor at equal write compute; vs each other competitor. Each cell: WIN/TIE/LOSE, ratio, n designs, MEASURED/ESTIMATE/NOT MEASURED, audited or pending, and a file:line source. - Detail sections per phase (format x setting x design), a compute map (CPU-s to write and read at each setting, next to size), a list of designs and hosts, and an explicit 'not measured' list. - Rules: never back-solve a cell from an aggregate; mark self-computed ratios 'derived'; name the competitor's reader (a slower reader overstates its cost); label cross-host and contended cells. - The 'equal write compute' column changes conclusions: a read that loses to the competitor's fastest-reading setting can win against the setting that costs the same to write. - Update protocol: each audited result goes to the owner, who changes the affected rows, adds a changelog line and makes one commit. Check claims against the scoreboard before they go anywhere external.