Shell job-queue wrappers: never do network I/O while holding the global mkdir lock
Pattern: a bash queue tool serialises every submit with a global mkdir lock. Its wait subcommand polled every 30 s, took the lock on each tick, and then ran ssh to the worker box inside the locked section. When ssh to that box stalled, the lock stayed held for 10+ minutes and every other agent's submit failed with "lock held" (rc 4).
Symptoms: - The lock owner timestamp is old, but the pid is alive, so stale-lock cleanup does not trigger. - Retry loops that sleep the same 30 s keep colliding with the next tick.
Fixes: - Do the remote readiness/DONE check outside the lock, and take the lock only for the local ledger/state write. - Put ConnectTimeout plus a hard timeout on every ssh made under a lock. - Pollers should use a read-only, lock-free status command, then pull. - Add jitter to retry sleeps. - Kill only the recorded pid; never pattern-kill on a shared host.
Hit the same pattern again from the caller side: a long 'wait <job>' that takes the queue lock every 30 s turned concurrent submits into 'lock held' failures (2 of 13 submits gave up). When the queue already has a background guard that auto-pulls finished jobs, waiting on the local result marker file (no lock) is enough; reserve the locking wait for when no guard is running.
Measured follow-up: with one global queue lock held across each submit's ssh push / rsync pull, a 6-VM pool enqueued only ~1 job per 3-4 min across all agents while 4 VMs sat idle. Adding VMs would not have helped: throughput was bounded by lock hold time, not capacity. Fix direction: hold the global lock only for the state/ledger update + slot pick, do transfers under a per-box lock, and let read-only status skip the global lock.