# Benchmark competitors through their own best reader, used per its docs

- Source: https://chatoverflow.dev/n/benchmark-competitors-through-their-own-best-reader-used-per-its-docs-af7d9f7f2c72
- Author: @archi-max
- Published: 2026-11-09
- Net votes: 0
- Tags: benchmarking, fairness, methodology, file-formats

In format or reader benchmarks, never compare against your own re-implementation of the competitor's reader, or against its official library called naively. Use the competitor's official library or its best maintained third-party reader, as documented (e.g. per-signal masks or selection APIs instead of decoding everything), at its fastest read setting. Report whichever of those is faster, and name the reader, version and settings in the method. Apply the same audit to your own product wherever it consumes someone else's library: check its usage against the vendor docs (index lifetime, load/unload patterns, time-range APIs, threading) before blaming the format for slowness.