Choosing a periodic checkpoint interval for debugging failing sim runs: watch phase luck and post-failure work that scales with window length

Setup: long RTL simulation runs take a periodic checkpoint every P of sim time. When a run fails, debug restores the last checkpoint before the failure and dumps only the window from there to the failure, instead of rerunning with full waveform dumping.

What a sweep over P (1, 5, 10, 50 ms on a ~45 ms failing run) showed: - In-run overhead falls roughly as 1/P: +3.4%, +0.8%, +0.5%, ~0. - Phase luck can make a large P look as good as a small one. At P=5 ms a checkpoint happened to land 0.3 ms before the failure, the same window as P=1 ms. Quote the worst case (window ≤ P), not the measured one, or sweep failure times. - If P is longer than the time to the failure, there is no checkpoint and you fall back to a full rerun. Pick P well below typical failure times. - Post-failure analysis can scale superlinearly with the window. Our root-cause walk took 33 CPU-s on a 0.3 ms window and 700 CPU-s on 5.3 ms (15x the bytes, 22x the time), enough to make P=10 ms worse than the old full-dump flow. Bound the analysis to a fixed slice before the failure and widen only on a miss. Then a larger P costs only the dump of the window, not the analysis. - Break-even failure rate = in-run overhead per run / per-failure saving. It is very sensitive to the per-failure path, so measure each stage (restore, dump, convert, walk) separately.

Run all P lanes in the same round, with rounds repeated, so host noise hits every P equally.