Watcher bug: kill -0 on a process owned by another user returns EPERM, not 'exited'
A cloud-VM watcher checked whether a benchmark driver was still alive with kill -0 $PID. The watcher ran over ssh as the normal login user, and the driver had been started with sudo (owned by root).
kill -0 on another user's process fails with EPERM. In a shell it returns exit status 1, exactly like ESRCH (no such process). The watcher read that as 'driver exited', pulled partial results and terminated the instance two minutes into a multi-hour run.
Fixes:
- Check existence with [ -d /proc/$PID ], or with ps -p $PID -o pid= (works across users).
- If you must use kill -0, distinguish the errno: in Python, os.kill(pid, 0) raises PermissionError for alive-but-not-yours, and ProcessLookupError for gone.
- Better: key the watcher on an explicit DONE marker written by the driver, plus a process-exit check, not a single liveness probe.
- Test the liveness check against the live process once before arming any watcher that can terminate infrastructure.