Instant 'stop' for long Python pipeline stages: run each stage in a killable subprocess group
Python threads can't be killed, so a job runner that executes stages on worker threads can only stop between stages. Fix that worked well: run each stage as subprocess.Popen([sys.executable, '-u', __file__, '--child-stage'], stdin=PIPE, stdout=PIPE, stderr=STDOUT, start_new_session=True), send the stage spec (plus a JSON snapshot of the parent's mutable config module) on stdin, and stream stdout into the job log. Stop = os.killpg(proc.pid, SIGKILL) which also kills ffmpeg/yt-dlp grandchildren (they share the new session/process group). Measured ~0.2s stop latency, ~0.1s per-stage overhead when heavy imports (torch) are lazy. Gotchas: (1) after a kill, delete the stage's outputs, since a half-written file (e.g. ffmpeg output) can look 'complete' to skip-if-exists logic; for per-item outputs delete only the newest file written since the stage started. (2) decode the pipe with codecs.getincrementaldecoder('utf-8') so multibyte chars split across os.read chunks don't garble; treat \r as newline and filter tqdm bars. (3) a stage waiting on a threading.Lock should poll lock.acquire(timeout=0.2) and check the cancel flag, or Stop won't be instant there. (4) side benefit: model memory is freed when each stage process exits.