VCS: a non-blocking checkpoint by fork() from VPI + $save in the child works only without -ucli
Goal: save a VCS simulation (W-2024.09) to a file without pausing it (a blocking save cost ~1.3 s each). Approach that works: build with an extra top module saver; reg [8*1024-1:0] name; reg go=0; always @(go) $save(name); endmodule (-top tb -top saver). From a VPI cbReadOnlySynch callback, fork(); in the child close the control pipes, redirect stdout/stderr to a log, schedule a callback at t+1 that puts the file name into saver.name and toggles saver.go (vpi_put_value), and another at t+2 that calls _exit(0). The parent returns in ~2 ms and keeps simulating. Restore with simv -r <file> (UCLI restore refuses $save images: UCLI-086). The restored image carries the child's VPI state, so guard every callback with a getpid() check against the recorded child pid.
Gotchas:
1. Only plain simv works. With -ucli -ucli2Proc the child prints the UCLI stop/finish/prompt lines and writes no save, and the parent's simulation gets SIGTERM ~0.5 s later. With single-process -ucli (multithreaded) the child deadlocks on a futex or the parent exits 255. Giving the child a silent stdin pipe did not help. So pick per run: UCLI checkpoints or fork-save.
2. $save runs at the end of its timestep, and cbEndOfSave never fired for it here: exit from a callback one step later, not from cbEndOfSave.
3. VCS compresses saves through an sh -> lz4 grandchild. Count the child's CPU with a cgroup (systemd-run --user --scope), not getrusage: per save it was ~1.4 s CPU in the cgroup vs ~0.17 s by getrusage.
4. Measured on 6.5M cycles: every 25K cycles, fork-save added +0.4% wall and blocking $save +103%. ~120 MB PSS per child; it takes no extra licence seat (lmstat).
Review notes from testing this pattern (VCS W-2024.09, small design): (1) In the child, return out of any request loop right after the fork. A child that keeps serving queued requests forks grandchildren and saves under the wrong name. (2) Exit 0 from the child says nothing: wait for its own children first ($save runs sh/nm helpers; a save took about 2-3 s wall even though the _tmp image reached full size in under 0.7 s), then stat the image and grep its log for '[VCS_SAVE_RESTORE] ... Error'. (3) Put a deadline on the child; a SIGKILLed save leaves <file>_tmp.save_in_progress and <file>.FILES/ behind, but never <file>. (4) If the parent sim ends while a child is saving, killing the process group loses that save silently; report outstanding children on exit.
Caveat found in review (VCS W-2024.09): don't point the fork child's writable fds at /dev/null to protect the parent's files. $save records each open $fopen file by what its fd points to (<save>.FILES held '17.null' instead of '17.out.txt'), so a run restored with 'simv -r' sends all its $fdisplay output to /dev/null and never creates out.txt. With the fds left alone, the restored out.txt was byte-identical to an uninterrupted run. If you want isolation, reopen each fd's /proc/self/fd target privately at the same offset instead. Also: restore reads <name>.FILES by the -r name, so renaming image and .FILES separately is not atomic as a pair.
Follow-up on inherited file descriptors in the fork child, measured on W-2024.09. To keep the child from writing into the parent's files, I first dup2'd every writable fd (3 and up) to /dev/null right after fork(). Don't do that: $save records each $fopen'd file by its current target, so it recorded /dev/null and the restored run (simv -r) wrote nothing to those files. What works: in the child, give each writable regular file its own open file description. Take the path from readlink /proc/self/fd/N and open it again with the same access mode, no O_TRUNC and no O_APPEND. Check that st_dev/st_ino match, lseek to the offset you read from the inherited fd, then dup2 it over N. This way the child never moves the parent's shared offset, and the restored run wrote the $fopen file byte-identical to an uninterrupted run. Leave read-only fds alone, because $save reads simv.daidir/dsa.sdb through one. Pipes, sockets and ttys can still go to /dev/null. Also, don't call fflush(NULL) in the child: that flushes its copies of the parent's stdio buffers into the shared files. Flush only the child's own stdout/stderr. One more finding: $save writes <name> plus a <name>.FILES directory. Both restore fine through symlinks, so you can commit a re-save atomically. Write each save into its own folder, make <name> and <name>.FILES fixed symlinks through a 'cur' link, and switch with a single rename() of a new symlink over 'cur'.
Follow-up (VCS W-2024.09): a layout that held up in review. The child saves into <file>.d/<pid>/img. <file> and <file>.FILES are fixed relative symlinks to <file>.d/cur/img and cur/img.FILES, and the commit is a single rename of a fresh symlink over cur. A crash at any step leaves a matching image/.FILES pair, and 'simv -r <symlink>' restores fine. For $fopen files, give each writable regular fd in the child its own description: reopen the /proc/self/fd target with the same access mode, no O_TRUNC, at the same offset, and check the inode matches. Restored output was then byte-identical, append mode included. Caveat: a save is now three entries, so moving or 'cp -r' of only <file> and <file>.FILES copies dangling links and the restore fails.