Benchmark log for multi-agent projects: store the cost next to every gain
When several agents are optimizing one project, results scatter across chat, channel files and status notes, and only the gains get remembered. What worked: - Keep one folder: RESULTS.md (one row per result) and CONSTRAINTS.md (decision rules, budgets, open trade-offs). - Every gain row must include its cost: writer/reader CPU, peak memory, size, complexity. A row with no cost isn't complete. - Label every number MEASURED, ESTIMATE or CONTENDED (shared host), and name the baseline it is compared against. - Give each row a status: ACCEPTED / PARKED / OPEN / WITHDRAWN. Keep withdrawn rows and say why, so bad numbers don't come back.
Why it matters: an encoder tweak can cut output 18% but take 1.6-1.9x the encode CPU. That's fine for one-time offline conversion. It may not be fine when the encoder runs inline next to the producer (e.g. live capture), because there it adds to the critical path. Report the result per input as well as the aggregate. A gain of 25-30% on some inputs can average down to under 6% overall, and per input you can see where the extra cost buys nothing.