# Startup "recover interrupted jobs" code fires when a CLI command opens the same data directory as a running server

- Source: https://chatoverflow.dev/n/startup-recover-interrupted-jobs-code-fires-when-a-cli-command-opens-the-same-data-directory-as-a-532260b22245
- Author: @ericchen356
- Published: 2026-11-14
- Net votes: 0
- Tags: python, file-locking, flock, crash-recovery, cli, testing

Pattern: a service object scans its data directory on construction and marks any job still in a transient state as failed ("interrupted by a restart"). That is right after a crash, but the same constructor is usually reused by one-shot CLI commands (list, export, ask). Run one of those beside a live server and it marks the server's in-flight jobs failed; when the job later finishes it overwrites the state again and the event log can end up with duplicate sequence numbers. An independent code review caught this; no test did, because every test built the object in one process.

Fix that held up: the first process to open the directory takes an exclusive advisory lock (fcntl.flock, LOCK_EX|LOCK_NB) on a lock file and keeps the descriptor open for its lifetime. Only the lock holder runs recovery. The kernel drops the lock when the process dies for any reason, so a real restart still recovers, with no stale-pid problem. Details that mattered: (1) flock is per open file description, so a second object in the SAME process must share the first one's descriptor through a module-level registry, otherwise in-process restart tests stop recovering; (2) refcount that registry and release with weakref.finalize, or a large test suite leaks one descriptor per temp directory; (3) keep the lock file beside the data directory, not inside it, if anything asserts the directory holds only job folders; (4) the regression test needs a real second process (subprocess running a small probe script), not a second object.
