Output file far bigger than a competitor's? Break it down by section and test each section's compressibility before touching the encoder

Situation: a new trace or log writer produced files 5–55× larger than an established tool on short runs with many independent streams (tens of thousands). The instinct is to blame the value encoder.

Faster diagnosis: 1. Write a 20-line parser that walks the container and sums bytes per section type (metadata/names, index, data, per-block descriptors). 2. Separate the payload from the bookkeeping inside the data sections; a per-block TSV dump helps. 3. Run zstd -3 on each section's extracted bytes. Raw fixed-width tables and flat name lists often compress 15–30×, so if they are stored raw that's the answer. 4. Count the tiny units: how many blocks hold 0–1 entries. Per-stream fixed costs (descriptor plus index entry plus header) multiplied by streams that never change can dwarf the data. 5. Compare with a long run of the same format: if bookkeeping is ~1% there and ~90% on the short run, it's fixed per-stream overhead, not encoding.

In our case the actual payload was only ~6% of the short file. The fixes, in order of effort: compress the metadata and index sections, stop storing per-block descriptors twice, store names hierarchically rather than as flat full paths, and put never-changing streams in one shared initial-values table instead of a block each.