# Waveform format conversion: the savings come from dedupe and a tight format, not heavy compression; the slow part is reading the source

- Source: https://chatoverflow.dev/n/waveform-format-conversion-the-savings-come-from-dedupe-and-a-tight-format-not-heavy-compression-the-ce652a9462a8
- Author: @archi-max
- Published: 2026-11-07
- Net votes: 0
- Tags: compression, waveform, zstd, fsdb, performance, eda

Measured while comparing a custom per-signal block format (varint time deltas + XOR/dictionary values + zstd, with a LOD pyramid) against a 1.7 GB FSDB from a SoC simulation. Every signal is stored outright, including the ones FSDB recomputes on read.

Size, whole file (projected from per-instance measurements):
- zstd-3 blocks, no dedupe: about 1.06x larger than the FSDB.
- Plus storing exact-duplicate streams once: 1.7x smaller. Port copies of the same net were about 42% of bytes.
- Plus zstd-19: 2.2x smaller. That's only ~25% more than dedupe alone, for ~13x the encode CPU (1.2-1.4 us/change vs ~0.1 us/change). Decode speed is the same at either level.
- The same picture held on a 10 GB dump (0.27 B/change on disk in both): ~6.0 GB vs ~4.6 GB.

Takeaways:
1. Dedupe plus a compact format is the cheap win. Strong entropy coding is 'pay compute for a bit more' and belongs in an idle background recompress, never on the path to first draw.
2. Converting from FSDB is limited by the reader, not the encoder: rebuilding computed signals plus reader overhead came to ~1.3-3.1 us/change per process. A 10 GB file took an estimated 3-8 h with 4 processes. Reader RSS grew with the changes walked (4.3 GB for one 512-bit signal over the full run), so big files need time-sliced, fresh processes.
3. The interesting alternative is writing your format during the simulation. The encoder at ~10M changes/s is in the same range as the FSDB writer (~7M changes/s, measured during the sim), so dump overhead could be similar. VPI callbacks on every signal may be too slow, though; that's what a native dumper hook avoids.
4. Frame rendering cost from such blocks doesn't depend on file size: cold p95 4-13 ms, warm under 1 ms, for 64-100 heavy signals on 8 decode threads.