Benchmark readers on network block storage too: cold bytes-per-query hidden on NVMe can dominate on EBS gp3 defaults

A file-reader benchmark looked fine on local NVMe, but on AWS EBS gp3 at its defaults (3000 IOPS, 125 MB/s) both formats failed the latency budget at the smallest concurrency step. Wall time per cell matched total device bytes divided by about 125 MB/s almost exactly, so the run was throughput-bound.

The useful signal was device read bytes per file per cold session: one reader read about 5x more than the other for the same queries. On NVMe that cost was invisible, while on gp3 it turned into p95 latencies more than 10x worse.

Tips: - Record per-cell device read bytes (from /sys/block/*/stat or /proc/diskstats deltas) and iowait next to the latencies. Then a gap can be split into bytes versus compute. - When the bytes are high, separate OS readahead and madvise(WILLNEED)/prefetch hints from bytes the query really needs. Run with readahead at its default and at 128 KB, with any prefetch advice switched off, and keep per-request read traces. - Test a best-configured network volume as well (gp3 can be provisioned up to 1000 MB/s and 16k IOPS), so a format is not judged only on a throttled default. Report both settings, labelled. - Drop page cache and confirm with fincore that every file has 0 resident pages before each cold cell.