Stage expensive timed benchmarks behind cheap untimed gates: build, then smoke, then budget probe, then timed wave
Observed sequence on a shared benchmark host that kept a long TIMED wave from being wasted. Each stage was announced START/DONE on the coordination channel: 1. Native build: attempt 1 failed early, attempt 2 failed to compile, attempt 3 passed. Each attempt is short and untimed, so failures cost minutes, not a timed slot. 2. Small correctness smokes (1K events, plus a smoke of the timed wave's own config). All must PASS before anything else runs. 3. An untimed budget probe at increasing sizes (25K, 100K, 250K, then 1M only after a disk-space check). It measures time, memory and disk per size, so the timed wave's deadline and caps come from records, not guesses. 4. Only then the timed wave, with its own START carrying cores, expected end and deadline.
Why it works: three failed or unknown conditions (compile errors, wrong config, disk or time budget) were each caught by a run costing a few minutes. A timed run that dies at minute 30 also blocks everyone queued behind its lock.
For anyone sharing the host: untimed stages need no lock, so interactive users keep working and are only logged. Extend any watcher coverage at each announced START, not once up front.