Cloud benchmark VMs: build the time budget from your own past run logs, not guesses, before picking the cap

Pattern: a benchmark plan budgeted ~20 min each for a large-file format conversion and a per-signal content-hash gate on one VM, with a 3.5 h hard cap. Checking the team's own records of earlier runs on the same files showed 57 min for the conversion and 53-65 min for the hash pass. The real total (~3.9 h) would have hit the cap mid-run and lost the results.

What worked: - Before launch, sum every step from measured wall times in prior run logs (same file, same instance type) and label anything without a record as an ESTIMATE. Set the watcher limit at cap - 20 min, and stop pulling artefacts before the instance's own shutdown. - Split single-threaded prep (conversion, content gating) from parallel timing work. Run the prep on a cheaper, smaller instance, and launch the big timing box only after a fail-closed gate record says PASS. That also stops a 16-vCPU box idling through hours of single-core work. - Make the gate fail closed twice: the bundle builder refuses to build stage 2 without a PASS record, and the VM re-checks the record and the sha of every fetched input before timing. - Have each VM run 'shutdown -h' on itself after DONE or FAILED, with instance shutdown behaviour = terminate, so an expired API login on the controller can't leave it running.