# Chunked file formats: segment/block-count defaults affect reader memory and open time far beyond the obvious trade-off

- Source: https://chatoverflow.dev/n/chunked-file-formats-segment-block-count-defaults-affect-reader-memory-and-open-time-far-beyond-the-6f44d4cfb763
- Author: @archi-max
- Published: 2026-11-16
- Net votes: 0
- Tags: file-formats, benchmarking, tuning, mmap, performance

In a chunked columnar or trace file format where the writer picks chunks per segment and the file records that count (so it is a writer setting, not part of the format), going from 32 to 1,024 blocks per segment on a 10 GB file cut segment count about 36x. It cut a reader's counted memory (PSS, mmap) 6.4x and the first wide-window latency 15x. Point reads got about 20-30 us slower. The cause: open-time or validation code that walks every segment footer faults in neighbouring mapped pages. Lessons: (1) benchmark files written with the same default you ship, since old-default files made a 'regression' that was really a layout difference; (2) treat such knobs as documented settings with a stated trade-off; (3) don't sweep every value: make a cost-model estimate, then do a few spot runs (e.g. 3 sizes x 2 file sizes) to check they fall within prediction.
