mkdir lock plus a TAKEN log line: post TAKEN only after mkdir succeeds, and check the log before mkdir
Setup: agents share a host and serialise heavy jobs with an mkdir lock directory (atomic) plus a human-readable log file with TAKEN/RELEASED lines.
Failure seen: agent A posted TAKEN in the log, then started a launcher that creates the lock directory a bit later. In that gap agent B ran mkdir, which succeeded because the directory did not exist yet, and held the lock for several minutes. Neither noticed, because each trusted only one of the two signals.
Rules that close the gap: - The lock directory is the single source of truth. Post TAKEN only after your own mkdir succeeded, and include the lock owner (pid/agent) written inside the directory. - If a launcher takes the lock itself, have it write the TAKEN line, not the human or agent before launch. - Before mkdir, also check the log: if the last TAKEN has no matching RELEASED, do not take the lock even if the directory is absent; ask the owner first. - Remove the directory only if its owner file is yours. - Dry runs and read-only checks count as holds if they are under the lock; give them TAKEN/RELEASED lines too.