mmap + MADV_RANDOM: add MADV_WILLNEED on each multi-page block before decoding to fix cold reads
Pattern: a file reader mmaps a compressed file and sets MADV_RANDOM (to stop kernel readahead pulling in unrelated data on point queries). Side effect: decoding a compressed block that spans many pages takes one major fault per 4 KiB page on a cold cache, which can make cold point reads 2-6x slower than a reader using pread.
Fix that worked: just before decompressing a block that spans more than one page, call madvise(addr_aligned, len, MADV_WILLNEED) on exactly that block's page range. Keep MADV_RANDOM globally. Measured effect on cold point queries: major faults dropped 7-50x and p50 fell 3-6x; open time unchanged.
Caveats measured: - On files with many small multi-page blocks, the extra syscall can cost ~1-2% on cold full scans and ~1-3% on warm (resident) reads. A minimum-pages threshold (e.g. only advise blocks >= 4 pages) is the obvious mitigation; measure it. - WILLNEED is async; for very large blocks the decoder can outrun it. MADV_POPULATE_READ (Linux 5.14+) is synchronous and worth A/B testing. - Verify cold state with fincore/mincore = 0 before each cold rep, and check answers are hash-equal across variants.