Python: kill subprocess children when the parent is SIGKILLed (PR_SET_PDEATHSIG via preexec_fn), plus the race and thread caveats

Problem: a Python scheduler starts worker processes with subprocess.Popen(..., start_new_session=True) so that it can kill each worker's process group. If the scheduler itself gets SIGKILL or is OOM-killed, no Python cleanup runs and the workers keep going. Neither atexit nor signal handlers help with SIGKILL.

What worked on Linux:

import ctypes, signal, os libc = ctypes.CDLL(None, use_errno=True) PR_SET_PDEATHSIG = 1 parent = os.getpid() def child_setup(): signal.pthread_sigmask(signal.SIG_SETMASK, set()) # undo the parent's block libc.prctl(PR_SET_PDEATHSIG, signal.SIGKILL) if os.getppid() != parent: # parent already died before prctl ran os._exit(1) Popen(cmd, start_new_session=True, preexec_fn=child_setup)

Caveats: 1. Race: if the parent dies between fork and prctl, the signal never fires. The getppid() check closes that window. 2. PDEATHSIG fires when the parent THREAD that forked the child exits, not the parent process. Start the workers from a thread that lives as long as the process (the main thread), or children die early. 3. preexec_fn is unsafe if the parent has other threads (CPython documents possible deadlocks) and is ignored in subinterpreters. Keep the scheduler single-threaded, or use a small exec wrapper that calls prctl, then exec. 4. It has to be the last parent: a worker that is a shell wrapper dies, but the wrapper's own children don't. In tests a bash stand-in's sleep 30 survived. Make the real worker the direct child (exec), or kill the group too. 5. To make your own handlers race-free, block SIGTERM/SIGINT around Popen and the step that records the child's pgid, then unblock. Otherwise a signal can arrive after the child exists but before you know its pgid. The child inherits the blocked mask, so unblock in preexec_fn as above. 6. Linux only; macOS has no PDEATHSIG. There you need a watchdog (for example kqueue EVFILT_PROC NOTE_EXIT in a helper, or a pipe whose EOF tells the child to exit).

Test: start the scheduler and its workers, kill -9 the scheduler, and assert that none of the recorded worker PIDs are alive after about 1 s.