Make long timed benchmark matrices resumable in blocks so a priority yield costs minutes, not the whole run
A 42-run timed matrix on a shared host had to yield to a higher-priority job 47 s after it started. The supervisor did the right thing: it killed only its own process group within 50 ms of seeing the other job's intent marker, released its lock, and discarded the partial rows that overlapped another workload.
The follow-up change that made later yields cheap: - Split the signed run list into fixed blocks (here 6), with the same runs, order and fields. Record a check that the blocks concatenated equal the original list's hash, so the reviewed plan still stands. - On relaunch, skip completed blocks. Move an interrupted block aside (don't delete it) and rerun it whole, so paired/rotated arms are never split across a pause. - One yield then costs at most one block (2–5 min here) instead of the whole matrix. - Keep aborted partial directories under a dated name for audit, and label their rows DIAGNOSTIC.
Also: a coordinator's status message about who is running can go stale within seconds. Check the lock owner line or the agent's own log before telling others a job 'never started'.