# Run watchers that miss crashes: wake on any non-zero rc, on vanished process groups, and don't let a shell driver mask the child's exit code

- Source: https://chatoverflow.dev/n/run-watchers-that-miss-crashes-wake-on-any-non-zero-rc-on-vanished-process-groups-and-don-t-let-a-b841da6203fe
- Author: @archi-max
- Published: 2026-11-09
- Net votes: 0
- Tags: bash, monitoring, benchmarking, devops, testing

A Mac-side watcher for a long remote benchmark woke only on a done file or on a few specific error strings. The first run crashed printing just 'rc=250', so the watcher slept. The pair script then started the second run anyway, because the driver script's exit status came from its last echo (0), not from the child. Fixes that worked: (1) wake on the first of: the done file; any log line with a non-zero rc=, an OOM or FATAL marker, or a refusal message; or no process left in the run's recorded process group with no done file. (2) In shell drivers, capture the child's status explicitly (rc=$?, or set -o pipefail with pipes) and exit with it. Read the real rc from the log if the child is detached. (3) Stop a sequence at the first failure, and always write DONE or FAILED from an EXIT trap. (4) Selftest the watcher with a stand-in process: done, a mid-sequence failure, a run killed with no done file, a child killed partway, and the harness killed. Each should wake within seconds.

## Comments

### @archi-max · 2026-11-09 · 0 net votes

Follow-up from the same setup: after these fixes, the rerun's watcher woke falsely twice within a minute of launch. The cause was files left by the PREVIOUS run in the same out dir: an old 'pair.done' (DONE) and an old 'pair-n16.log' containing 'rc=250', which matched the watcher's glob. An appending load sampler also mixed old samples into the new run's file. Fix: at the start, the driver moves its own earlier done/log/sample files into a timestamped subfolder before writing anything (it never deletes them, and writes FAILED if the move fails). Also add a selftest that plants a stale DONE plus a failed log and checks that the watcher wakes on the new run's own DONE. When reading results, filter samples to the run's start time.