Cold point reads via mmap: tiny reads can trigger ~150 KB read-around and hide index-size wins

Benchmarking cold 'open file + read one small record' latency over an mmap'd file on Linux, with the page cache evicted per file (fadvise DONTNEED, residency checked with fincore). We shrank the file's on-disk index, so open got ~10 ms faster, yet total cold latency didn't move. The reason: touching one ~11-byte record faulted in ~152 KB (38 pages). That is the kernel's mmap read-around on a page fault, which for file mappings is driven by the readahead window (read_ahead_kb), not just the 4 KB page you need. The point phase gave back everything open had saved. In another layout, the record's page happened to be inside what open had already pulled in, so the fault was nearly free.

Lessons: - Time the open (mmap + parse) and the point decode separately. Otherwise a win in one hides a loss in the other. Our first pass timed only the decode and gave the wrong conclusion. - Count the bytes actually read per query (e.g. /proc/self/io read_bytes, or mincore before and after), not just wall time. 'Small record, big I/O' points straight at read-around. - Candidate fixes: madvise(MADV_RANDOM) on the data region of the mapping, which disables read-around; or pread() the record's page into a buffer instead of touching the mapping. Re-measure the cold full sequential read afterwards, because disabling read-around can slow sequential scans. Apply it per region, not to the whole file. - Cold numbers depend on read_ahead_kb and the device, so record both with the results. - Use n >= 3 with paired per-rep deltas. Our size-optimised layout showed +0..+5 ms per rep with overlapping ranges, which is 'no difference detected', not a regression.