Two run-watcher traps: a pgid checked before setsid has run, and an EXIT trap that writes DONE when the script is killed
Two bugs I found while reviewing a 'wake on done / failure / process group gone' watcher for long batch runs.
1. The launch races the pgid check. A test helper started each fake run as python3 -c 'os.setsid(); os.execvp(...)' & echo $! and handed $! to the watcher as the process group id. The watcher's first pgrep -g $pid ran before python had called setsid, found no group, and reported 'run gone' at 0 s. On macOS this failed 3 of 3 runs; with a 0.5 s sleep after the launch it passed every time. Fix: after launching, poll until ps -o pgid= -p $pid equals $pid, then start watching. Also put a lower time bound on the 'gone' cases (e.g. took at least 1 s), or the same race makes them pass vacuously.
2. An EXIT-trap summary claims success after SIGTERM. A sequence script ran N=8 and then N=16, appended 'nX rc=R' after each step, and wrote DONE from trap fin EXIT when every line in the rc file was rc=0. bash runs EXIT traps on SIGTERM too, so killing the script during the second step wrote DONE with only the first line present. Any watcher keyed on DONE then reported success. Fix: require the expected number of rc=0 lines, or write a 'running' line before each step and replace it afterwards.
Also: a watcher that polls over ssh should cap its wall time or count consecutive ssh failures; otherwise an unreachable host makes it loop silently forever.
A third trap of the same kind: stale done/log files from an earlier run. A run script that moves those files aside in its first lines still races the watcher. The pgid exists as soon as setsid runs, before the script's first line, and the watcher checks at once. In a measured repro, a 0.3 s delay between setsid and the script (a launcher that cd's, sets env, or a loaded host) made the watcher wake on the old DONE 10 times out of 10. Launched directly, it never did (0 of 20). Fixes: have the launcher clear or move the old files before setsid, or start the watcher only once a file the run writes after its cleanup (for example its own sid/pid file) holds the new id. A test that plants stale files only exercises this if it adds a delay before the script starts.