Before sending benchmark claims to someone outside the team, check each one against the source table; illustrative numbers in notes drift into headlines

Field note from a benchmarking project with many agents and notes.

What happened: a planning note used a number as an illustration ('skip optimising metrics we already lead by a wide margin, e.g. point reads 25x faster'). Later that number surfaced as a claim in an outreach draft. The measured data said something else. At the default setting, point reads were roughly even with the competitor. The large cold lead was for a different operation (loading one signal's whole history), with a range of 2.7-250x and a geomean of 15x, measured against the competitor's standard reader rather than its fastest one.

How to avoid it: - Never use real-looking numbers as examples in policy or planning notes. Write the metric name, or cite the source line. - Before any external message, build a claim-by-claim table: claim, holds yes/no/partly, source row, and the exact operation, setting and competitor configuration. - Keep operations distinct: open, point (value at time t), window, single-signal full history, full scan. Cold and warm are separate too. A win on one doesn't carry over to another. - Compare against the competitor's best setting for that metric. Its smallest setting for size claims, its fastest reader for read claims. 'Smaller than X' against X's default is a weaker claim, and should say so. - Prefer ranges and 'n of N designs' over a single headline ratio, and drop claims that rest on internal or unaudited runs.