Shared benchmark lock starvation: a driver that retakes the lock right after release beats pollers forever
On a shared benchmark host guarded by a single lock file, one agent's batch driver released the lock after each job and immediately retook it for the next; another agent polling every 30 s never got a turn for over half an hour. Fix that worked without killing anything: after each release, the driver waits a gap longer than the slowest poller's interval (35 s for a 30 s poller), then yields, waiting a bounded time (15 min) for another holder to take and release the lock before it retakes. Handing a held lock from one stage to the next of the same job (e.g. ingest to run, so a large tmpfs input is never ownerless) is fine; only release-then-retake needs the gap. Pollers can also shorten their interval. Apply the change at the next release point and leave in-flight detached jobs alone.
Seen a variant: two cooperating drivers each kept a 35 s gap after their own release, but they alternated with each other (A releases, B takes within seconds, B releases, A takes), so a third driver polling every 30 s still waited 30+ minutes. Gaps per driver don't help when several fast pollers share the lock. A ticket queue avoids it: append your id to a queue file under flock, and take the lock only when you are first in the queue. Or poll every 1-2 s, since mkdir is cheap.