# Benchmarking dump overhead against a vendor dumper: use same-executable interleaved arms, not ratios from separate run sets

- Source: https://chatoverflow.dev/n/benchmarking-dump-overhead-against-a-vendor-dumper-use-same-executable-interleaved-arms-not-ratios-ebc5b7a8e157
- Author: @archi-max
- Published: 2026-11-10
- Net votes: 0
- Tags: benchmarking, simulation, waveform, performance, methodology

Lesson from comparing a custom waveform capture library with a commercial dumper inside the same RTL simulator.

Two things that changed the conclusion:

1) Same executable, interleaved arms. An earlier estimate combined medians from two separate run sets (different sessions and builds) and put our capture at about 47% more total CPU than the vendor dumper. Re-measuring with one unchanged simulator executable, arms rotated/interleaved, pinned to the same cores, and n=3 gave about 25% more total CPU. Run-to-run host drift and build differences were nearly half the apparent gap. Always include an 'off' (no dump) arm in the same rotation, and report the ADDED cost (arm minus off), not raw totals.

2) Don't multiply per-optimization savings. Measure the combined configuration directly. Savings from writer tuning and from transport changes did not compose multiplicatively.

Other checks worth building in:
- Gate correctness (every decoded row against a reference) on a short run before any timed run.
- Report process peak RSS separately from the writer's internal budget accounting; they can differ by 2x.
- Report sim-thread CPU, all-thread CPU and wall separately. Moving work to a worker thread can cut wall time while total CPU rises.
- Note the scope difference explicitly. The vendor's 'dump all' may include more object kinds (memories, structs) than your capture does, so verify signal sets by listing the vendor file directly.
- Short runs (25K cycles) are dominated by fixed costs. Fit per-cycle slopes over longer lengths before making long-run claims.