Lock file vanished: a cache dir "sweep leftovers" pass deleted another process's lock
Symptom: two processes sharing a per-key cache directory both ran an expensive job that an O_EXCL lock file (job.lock) was meant to serialize. A stale-lock takeover race looked like the cause, and hardening the takeover (a second takeover lock plus an owner re-check) did not fix it.
Actual cause: on open, each process swept its cache dir and deleted every file whose name was not on a known list (meant for leftover temp files). The lock was not on the list, so any process opening the same key deleted the lock the other process was holding.
How it was found: log the dir listing (readdirSync) at the moment of the lock attempt. The lock was missing although its owner was alive.
Fix: add lock files (and any takeover/sidecar locks) to the sweep's skip list. Also re-read the result cache right after acquiring the lock, because the previous holder may have finished in between.
Lesson: when adding a new file type to a shared dir, grep for every code path that cleans that dir.
Same trap with flock(1) and an age-based sweep. The sweep deleted lock files 'untouched for a day', but taking an flock never changes a file's mtime. So a lock created yesterday looks stale while someone holds it right now. After the unlink, the next locker creates a new file with a new inode and gets its lock at once (measured: 176 ms instead of waiting 3.6 s for the holder). flock excludes per inode, not per path. Fix: never sweep the lock files themselves (they are empty), or unlink one only after taking it with flock -n.