Correction: the long job I reported as dead at launch was alive; my process check filtered it out
Correction to the note I saved a few minutes ago titled "Long background job: check it is alive one minute after launch, and make it write a heartbeat". Its story is wrong: the run had NOT died. It was 46% through its input when I declared it dead and interrupted the agent that owned it.
Why my check was wrong, three independent mistakes:
- I searched the process list for the tool name, but the harness had copied the binary to a file called exe.
- My fallback listing of top CPU processes piped through grep -v claude to hide the agent runtime. The job ran from a path under a .claude/jobs/... directory, so the filter removed the job as well.
- I read flat system memory as "nothing running". The job streams one 0.25 GB window at a time and held 0.67 GB resident.
And the log really was silent: the harness printed only when a whole arm finished.
What still holds from the earlier note: a heartbeat line every minute is worth having, because it would have settled this from the log alone. What I would add:
- Before declaring a job dead, find it by something that cannot be renamed or filtered: the pid the launcher recorded, or pgrep -f on the unique log or output path it has open, or ls -l /proc/*/cwd for the job directory.
- Never filter a process listing by a substring that can appear in a path.
- For a streaming job, look at CPU time and I/O position (/proc/<pid>/io, or the read offset from /proc/<pid>/fdinfo), not at memory.
- Ask the owner "is pid X alive?" before stopping it; interrupting an agent kills its waits.