mmap readers on network block storage: kernel readahead around scattered faults inflates bytes; batch-prefetch metadata and MADV_RANDOM the data

An mmap-based file reader that touches small metadata records scattered across a large file (per-segment footers, index entries) can read far more bytes than it needs from cold storage, because every page fault pulls in a readahead window. Measured with /proc/self/io read_bytes per request on a 1 GB file: one cold session read 35 MiB at read_ahead_kb=16, 67 MiB at the default 128 and 154 MiB at 512. On local NVMe this is hidden; on network block storage (e.g. EBS gp3 at 125 MB/s) it dominates p95.

Turning off readahead alone (MADV_RANDOM on everything) cut bytes but made latency much worse (p95 207 ms vs 43 ms), because each scattered 4 KiB fault became its own network round trip.

What worked was one up-front MADV_WILLNEED, or a sequential read, over the small metadata region, plus MADV_RANDOM on the bulk data blocks. Bytes became independent of read_ahead_kb (~30 MiB per session) and p95 fell 7-13x.

Tips: - Measure bytes per request from /proc/self/io; check that per-process totals equal the device bytes. - Sweep read_ahead_kb to see whether bytes are readahead-driven. - Test default and provisioned network volumes under concurrency, not just N=1. A single client is latency-bound, so throughput limits only show with many clients. - Advise one contiguous region per call, so the VMA is not split (vm.max_map_count).