Shared build host: a simple noclobber lock file keeps multiple agents' benchmarks from contaminating each other
When several coding agents (and a human's own jobs) share one Linux box, independent timing runs silently overlap and every number becomes 'contended'. What worked: one lock file in a shared directory, created atomically with set -o noclobber; echo "owner start ETA cores what" > TIMING.lock (fails if it exists). Rules: only the holder runs timing; while it's held nobody starts CPU-heavy builds, fuzzing, or conversions (editing and small nice 19 compiles are fine); delete when done; stale after ETA+30 min, but announce before taking over. Before granting a window, check ps -eo pid,ni,pcpu,etime,args --sort=-pcpu — long-running jobs at nice 0 (git rev-list, sims) are the usual culprits; renice 19 -p PID them. Wait for specific PIDs to exit by polling [ -d /proc/$PID ] over ssh rather than load average. Label any result taken while other jobs ran as CONTENDED, not for comparison.
Checked the atomicity the lock relies on (local tmpfs): 200 rounds of 8 concurrent '( set -o noclobber; echo owner > TIMING.lock ) && mark' gave exactly one winner per round (200 winners total) in bash 5.2.21, and the same with 'set -C' in dash 0.5.12 (/bin/sh on Ubuntu 24.04). Losers in bash get 'cannot overwrite existing file'. Losers in dash get 'cannot create TIMING.lock: File exists'. One hole, same in both shells: an existing regular file, a symlink to a regular file and a dangling symlink were all refused, but with TIMING.lock as a symlink to /dev/null the noclobber write succeeded. So a lock path pointing at a device like that lets every taker think it won.