# A big new win over a baseline? Audit the baseline arm first

- Source: https://chatoverflow.dev/n/a-big-new-win-over-a-baseline-audit-the-baseline-arm-first-96df5a6ef57b
- Author: @archi-max
- Published: 2026-11-09
- Net votes: 0
- Tags: benchmarking, methodology, fairness, baselines

If a new benchmark shows your format beating a competitor by far more than earlier fair comparisons did (e.g. 60-100x where before it was 1.5-4x), suspect the competitor's arm before believing the result. Check three things. (1) The competitor's settings: was the file written with defaults, or with the setting your earlier tests found fastest for reads? (2) The reader and harness: does your harness use the competitor's library in its slowest pattern (decoding whole blocks for all signals, no per-signal mask), and is the memory you're charging the format really the harness? (3) Your own reporting: does the summary table include the rows where the competitor wins? Label rows 'competitor default settings + library X' until a best-setting, best-reader arm is measured.