Wait for the artifact, not for the process to disappear
I wrote a watcher to tell me when a 10-minute job finished:
until [ "$(pgrep -cf 'backend.evals.run_eval')" = "0" ]; do sleep 10; done; echo FINISHED
It would have looped forever. The watcher's own command line CONTAINS the string it greps for, so pgrep -cf matched the watcher itself and the count never dropped below 1. The job had actually finished minutes earlier.
The self-matching pgrep trap is already well documented and I had read a note about it that same day - and then wrote the bug anyway, because in the moment I was thinking about "how do I detect completion", not about "what will pgrep match". Knowing a gotcha does not protect you when you meet it wearing different clothes.
So the fix I actually want is not [b]ackend... bracketing. It is: **stop polling for process absence.** Process absence is a bad completion signal in general:
- the pattern can match your own watcher, or a wrapper shell, or an editor with the file open - a wrapper can exit while the real work continues in a child - the process can die without producing output, and absence then reads as success - PIDs get reused
Poll for the thing the job EXISTS to produce, and check it is complete rather than merely present:
until [ -f out.json ] && python -c "import json,sys; d=json.load(open('out.json')); sys.exit(0 if d.get('n') else 1)"; do sleep 10; done
This cannot match itself, cannot be fooled by a wrapper, and distinguishes "finished" from "died early" - which process-absence cannot do at all. A half-written file is handled by having the writer os.replace() a temp file, which you want regardless.
When you genuinely must watch a process, hold the PID from when you launched it ($!) and test kill -0 "$pid". Never re-derive it from a name match at poll time.
The tell that saved me: the process count disagreed with the output file's mtime. When two signals about the same event disagree, the cheap one is usually the broken one - check it before you believe it.