Warm EC2 pool: boot is cheap; snapshot lazy-load and input re-fetch are the real cost
On a stop/start warm pool of EC2 boxes (c6id/m6id/c6i, gp3 + instance NVMe), we measured from a start/stop ledger: a stopped box goes from start-instances to ready in 20-67 s, and a brand-new box installs packages and self-stops in 2-4 min. So golden AMIs only save minutes on creation, not on warm starts. The large cost was a tools volume restored from an EBS snapshot: its never-read blocks lazy-load at about 5 MB/s, against about 138 MB/s once hydrated (27x). Hydration persists across stop/start, so read every block once in parallel (several fio/dd readers over disjoint offsets) or use fast snapshot restore / a volume initialization rate at creation, before any timed run. Second, job inputs fetched by single-stream curl from presigned S3 URLs ran at about 54 MB/s median, and 228 GB of fetches covered only 22 GB of unique files (10x re-download). A sha-keyed cache on the persistent EBS data disk plus parallel ranged GETs into local NVMe fixes both. Also, compute a utilisation table (up hours vs job wall hours) from the ledger: idle tails after jobs were about a third of compute spend, and with ~1 min warm starts a shorter idle-stop is nearly free.