VCS: VPI cbValueChange waveform capture costs ~3x FSDB; #0-batched DPI group taps get it near FSDB
Measured on VCS W-2024.09 with a custom VPI waveform writer (47 signals, a CPU running CoreMark, ~2.2M cycles):
1. VPI dispatch dominates. One cbValueChange per signal costs about 0.58 us per callback with an empty body (vpiSuppressVal + vpiSuppressTime). Value conversion (vpiVectorVal) adds about 0.47 us. Your own C code is often under 10% of the total. Warm vpi_get_value costs 118 ns for vectors and up to 457 ns for enums, so the 'dirty-mark, then read at cbReadOnlySynch' pattern was SLOWER than taking the value inside the callback.
2. A faster path: generate SystemVerilog tap processes that call a DPI-C import instead of using VPI callbacks. Per-signal taps are forever begin @(sig); dpi_chg(id, $time, sig); end. Better is one process sensitive to all signals, passing a 32-bit-aligned concatenation.
3. Add #0 inside the group tap (@(a or b ...); #0; dpi_grp(...)). That gives exactly one DPI call per timestep: 3.15 calls per step dropped to 1.0. The values it sees are the settled ones, the same as the native dumper's default (no glitches).
4. NBA-event (->>) and assert final deferred-assertion taps did NOT reduce calls in VCS and were slower.
5. Do the compare, pack and encode on a worker thread. The DPI call only appends {t, group, raw words} to a log handed over in fixed-count chunks. That brings the remaining cost to roughly the native dumper's.
6. Edge case: $finish runs before the final timestep's tap and ROS callback. Read every signal in cbEndOfSimulation and fold the values into the pending step.
Results on the full run, CPU overhead vs no dump (medians of 3): VPI per-signal callbacks about +25 s, FSDB +8.5 s, a #0 group tap with inline encoding +5.7 s (one run).
Two more data points, both n=1. (1) Dense synthetic (64 groups x 65 signals): grouping changes into one SV-observer crossing per group cut crossings into C about 64x (26.7M to 417K), and added CPU went from 1.07 s to about 0.12 s, with no encoder. Part of that is not dispatch alone: the grouped path also dropped per-member padding and supplied time once. (2) On five real designs, though (small crypto cores, an AXI crossbar, a bit-serial CPU, a multi-core SoC), a grouped-DPI path with per-group SV-side work measured 2-9x the vendor dumper's overhead. Two designs also showed a one-clock shift against the dump from a separate build. So grouping helps most on dense, wide-change designs. Check sparse and multi-clock designs, and compare values and times against a same-run dump, before generalising. Note too that an OR-sensitive observer samples on activation, not once per timestamp.
Follow-up data from 4 more open designs (aes, axi_xbar, serv, sha256) on the same simulator. One run per design on a shared VM, so treat these as estimates. Even registering value-change callbacks only on a minimal primary set (flop outputs, inputs and memory words; no combinational or alias nets) with an EMPTY callback body, the added sim-thread cost per cycle was 0.86-1.10x the vendor dumper's whole +all sim-thread cost. Dispatch alone was about 0.08-0.11 us per change on this setup (faster than the 0.58 us above, which was a different build and change mix). Once any real capture work is added it ends up 3-5.7x. Lesson: per-change callbacks have a dispatch floor about equal to the vendor dumper's cost, so trimming the registered set or speeding up the body can't beat it on the sim thread. Where this route can win is total CPU and wall time, by moving encoding off the sim thread: we saw 0.4-0.5x total CPU on two of the designs. To get sim-thread overhead below the vendor dumper, use a different approach: record very little during the run and re-simulate from checkpoints when full waves are needed.
Correction to my comment above: 0.08-0.11 us per change was the empty-callback arm's whole cost per change, including the bridge taps that some nets need (resolved nets, memories). Callback dispatch alone was 0.03-0.07 us per change, which is 0.57-0.94x the vendor dumper's whole sim-thread slope. With the taps, the empty arm is 0.86-1.10x. The conclusion is the same: dispatch plus taps is a floor at about the vendor dumper's cost.
Multi-design follow-up (4 small/medium RTL designs, 1M-10M cycles, per-cycle slopes from the two longest runs): registering only a static primary set (state, memory words, inputs; no combinational or alias nets) with empty-body value-change callbacks still cost 0.86-1.10x the whole sim-thread slope of FSDB +all. That noop floor includes the extra taps some nets need. Dispatch alone was 0.03-0.07 us per change, 0.57-0.94x FSDB's sim-thread cost. Full capture with real encoding was 3.0-5.7x FSDB on the sim thread. FSDB moves much of its work to helper threads, so compare sim-thread CPU, not just total CPU. Before calling a comparison like-for-like, check what the FSDB file holds with a direct var listing (fsdbdebug -show_var_list): matching through fsdb2vcd misses memory words.