Cold-cache read benchmarks: check bytes read and major faults before blaming CPU

When a cold-cache read benchmark shows one file format or reader much slower than another on a single small query, record these per process before profiling CPU: read_bytes from /proc/self/io, the major fault count (getrusage ru_majflt), and user and sys time.

What I saw: a reader that mmaps the file answered one point query in about 90 ms cold, with almost no user CPU, while reading about 90 MB from disk (about 22% of the file) through about 800 major faults. The competing reader read under 100 KB for the same query. Dividing read_bytes by the fault count gave about 114 KB per fault. That is the kernel's mmap read-ahead window, so each scattered page touch became a large read.

What to check: - Compare read_bytes to the file size, and to what the query logically needs. If a single query reads a large fraction of the file, the problem is the access pattern, not decode speed. - Check whether read_bytes grows with file size or with the number of chunks the file has. That points to a lookup that walks every chunk instead of seeking straight to the right one. - Possible fixes: seek directly by time or offset; madvise(MADV_RANDOM) on regions you only probe; keep lookup metadata contiguous so it fits in a few pages. - Measure cold reads properly: use drop_caches on a dedicated VM, or fadvise DONTNEED per file plus a fincore check showing 0% resident on shared hosts, and use a fresh process for each cold sample. - Report open time separately. Here the open was cheap (a few ms, about 10 faults), and the whole cost was in the first query.