Verifying a waveform-capturing regression tool: independent ground truth plus negative controls
To check that a regression tool's stored waveforms are exact, don't compare its store against its own capture path. Rerun a seeded, stratified 10% sample (passing and failing runs, every test) with the simulator's built-in tracer (e.g. Verilator --trace VCD) for the same signals, and diff each signal's full change history bit for bit, at every change time, including x/z and the run boundaries. Find root-cause claims with your own bug-vs-fix first-divergence diff, not the tool's fan-in code. Add negative controls that must fail: corrupt one stored chunk, shift one change by one time unit, swap two runs. Otherwise a 100% match may be vacuous. Keep the stores you plan to audit: a burst VM's local disk vanishes at termination, and then the store has to be regenerated from pinned inputs (tarball sha256 plus repo commits) before you can audit it.
Applied this end to end (Verilator 5.052, ibex). What else mattered: (1) Retention drops runs from the store, so a fidelity sample of passing runs needs a pre-drop snapshot. Wrap the store's drop() and copy each manifest first (the chunk bytes usually stay until compaction), then read dropped runs through the tool's own reader. (2) Derive the expected stream from the VCD using the tap's documented sampling rule (posedge-settled sample per cycle, req/valid gating). Comparing against raw VCD changes 'fails' by design for gated payloads. Also count changes that appear only at negedge times, since a per-cycle tap can't see them. (3) Regenerating the run with identical code and params is itself a test. It exposed a parallel-triage race: tasks wrote a reference run's list file to the same path, so signatures differed run to run while serial runs were identical. (4) Check whether 'checkpoint windows' actually restore. Most divergences were before the first checkpoint, so the speedup came from knowing t_div. (5) Bug-vs-fix root cause without a graph: the earliest-diverging signal in the whole design must be the changed assignment (plus port aliases and comb consumers). Compare by hierarchical name, because two builds number VCD ids differently. (6) Don't stream VCDs through FIFOs into a lock-step Python reader: it deadlocked. Write files instead (they're small up to t_div).