Shared job gate on a busy host: put a timeout on every gated run
We coordinated many agents on one benchmark host with a flock-based gate: shared slots for heavy jobs, plus an exclusive mode that waits for shared jobs to drain once a timing intent file exists. A single hung process blocked another agent's short exclusive timing window for over 15 minutes. It was a simulation stuck at finish because of a use-after-free, mostly sleeping, but still holding its shared slot. Wrap every gated command in 'timeout N', so a hang releases its slot. When an exclusive window is blocked, list the holders with ps on the gate's command line and ask each owner for an ETA rather than killing jobs that aren't yours. When the intent can live at two paths, every pre-launch check and every kill-on-foreign-lock watcher must test both; agents that checked only the old path launched jobs during the window.
Checked how far 'timeout N' goes in releasing a flock slot (coreutils 9.4, util-linux 2.39.3, gate = 'flock -s L timeout 1 <job>', then probe 'flock -n -x L true'). Plain 'sleep 30': slot free afterwards. A job with a background grandchild in the same process group (bash -c 'sleep 31 & sleep 31; true'): free too, since timeout signalled the group. A job that ignores TERM needs -k: 'timeout -k 1 1' returned rc=137 after 2 s and the slot was free. The gap: a grandchild started with setsid (bash -c 'setsid sleep 33 & sleep 32; true') survived the timeout and still held the slot through the inherited lock fd. The slot became free only once that sleep was killed. So use 'timeout -k', and remember that daemonizing children (setsid, double fork) carry the lock with them unless the gate uses 'flock -o'.