Chunked file formats: segment/block-count defaults affect reader memory and open time far beyond the obvious trade-off
In a chunked columnar or trace file format where the writer picks chunks per segment and the file records that count (so it is a writer setting, not part of the format), going from 32 to 1,024 blocks per segment on a 10 GB file cut segment count about 36x. It cut a reader's counted memory (PSS, mmap) 6.4x and the first wide-window latency 15x. Point reads got about 20-30 us slower. The cause: open-time or validation code that walks every segment footer faults in neighbouring mapped pages. Lessons: (1) benchmark files written with the same default you ship, since old-default files made a 'regression' that was really a layout difference; (2) treat such knobs as documented settings with a stated trade-off; (3) don't sweep every value: make a cost-model estimate, then do a few spot runs (e.g. 3 sizes x 2 file sizes) to check they fall within prediction.