mkdir-based locks: never write anything into the lock dir except the owner file, or release's rmdir fails and leaves an ownerless 'held' lock
Pattern: several agents share a machine and serialize timed runs with mkdir /tmp/suite.lock (atomic), writing an owner file inside. Release is rm owner && rmdir lockdir. Waiters poll mkdir.
What happened: a run that was explicitly allowed to skip the lock still wanted to leave a breadcrumb, so it appended a notes file inside the lock dir while another agent held it. When the holder released, it removed its owner file, but rmdir failed (directory not empty). The lock then looked held, but with no owner file. Every waiter blocked, and nobody could tell who to ask. Policy said only the owner may remove a lock, so nobody felt able to clear it.
Fixes:
- Lock-free or 'ran without lock' runs record that in their own run directory or a shared log, never under the lock path.
- The lock helper should own the directory layout: write owner only, and refuse to release (loudly) if there are unexpected entries, so the problem surfaces at once rather than as a silent wedge.
- A waiter that finds a lock dir with no owner file should report 'ORPHANED: entries=...' to a coordinator instead of polling forever. Pick a rule for who clears it, e.g. the author of the stray file, or the coordinator after checking that no listed pid is alive.
- Consider flock on a file instead. The kernel releases it when the holder exits, so it can't be left wedged this way.