Review checklist for a 'warm stopped VM' run driver: seven failure modes a green fake run won't show
We wrote a driver for a reusable EC2 bench VM (start → pull inputs → run kit → push → DONE → Mac pulls → stop). A stubbed fake run passed all 40 checks, yet review found seven real bugs, because the stubs never modelled time, disk or leftover state. Check these explicitly: 1. Kit and download timeouts must be clipped to the time left before the in-guest shutdown cap; otherwise a long kit dies before writing DONE and results are lost. 2. The cost ledger must not stop counting when the Mac-side watcher gives up: the VM keeps billing until its cap or idle stop. Log an upper-bound cost, then the confirmed stop time. 3. Stale state from earlier runs (an unpulled DONE marker, a PID file whose number was reused) can block the idle stop on every later boot. Key state files by run id and validate them. 4. Root disk fills: large inputs need instance-store NVMe (mkfs each boot) or a free-space check plus cleanup after the pull. 5. Volumes restored from snapshots load lazily, so the first full read is slow. A first-boot integrity hash can look 'idle' and trip the watchdog; heartbeat during it, and don't count timings until one full read is done. 6. Billing starts at instance start, not at first ssh: arm the cost cap from start. 7. A kit exiting non-zero must make the driver exit non-zero, never be logged as done. Fake runs should inject: slow downloads, a full disk, leftover markers from a previous run, and a failing kit.