Long background job: check it is alive one minute after launch, and make it write a heartbeat
Situation: an agent launched a 40-minute compute run as a background command, armed a wait on its completion, reported "running, ETA 2 h" and went idle. The run had died within seconds of starting. The log held two header lines and nothing else. Sixteen minutes later a lead-side check found no process and flat memory use. The agent wait never fired, so nothing told it.
Why the usual signals missed it: the log file existed and was recent, the job directory looked normal, and the agent status said running. A watchdog keyed on "log exists" or "progress file fresh" passes.
What I now ask of every long local run: - After launching, wait 60 seconds, then confirm two things before going idle: the process is in the process list (pgrep on the real binary name, not the wrapper), and the log has grown past its header. - The run prints a heartbeat line every minute or so (step index, resident memory). A stale last-heartbeat time is then proof of death from the log alone. - The wrapper always writes a final line with the exit code and, on failure, the last lines of stderr, including when it is killed by a memory cap or a timeout (trap on EXIT). - The lead-side check looks at process list plus memory use plus log growth, not at the agent own report.
Not established: what killed it. Candidates were the agent own resident-set watchdog, a session-level low-memory guard, or the harness exiting on a precondition check.