Comparing file-format readers fairly: one pure query runner, seeded queries, a labelled source per column
Pattern for a page that compares how fast different readers or file formats answer the same reads, e.g. waveform dump formats.
1. Write the bench as a pure module that takes req(request) -> {result, ms, bytes} and clock. Every reader then answers exactly the same queries, and you can unit-test it with a fake reader: check the walk, the query count, that each scope is asked for only once, and the stats.
2. Generate the query times with a seeded PRNG (e.g. mulberry32), say N point reads uniform in the last 1% of the time span. Every run and every format gets identical queries.
3. Time the operations a viewer actually performs, not one synthetic number:
- open;
- walk the hierarchy to the items by name (array or generate scopes may come back flat or nested inline: handle both);
- first read, then the same read again (warm);
- point reads (p50 and p95);
- one full stream read (what drawing a row costs).
4. When a relay sits in between, record browser wall time and the reader's self-reported time side by side; the difference is the transport overhead.
5. Start a fresh reader process for each run so the open is cold for the reader, and say so in the page: the host's OS file cache may still be warm.
6. Results from someone else's benchmark go in their own column with source, file and date, never merged into yours. A format you can't run yet stays visibly pending, not blank.
7. If no marker or time is configured, default to the middle of the data. A first read at t=0 hides the cost of seeking.
Testing tip: a fake reader whose clock advances a fixed step per call makes the timing assertions deterministic.