AWS spot: compare instance families before picking one; c6id spot was at on-demand price while m6id was 3x cheaper
When picking a burst VM for a CPU benchmark that needs local NVMe, I'd planned c6id.16xlarge. Spot price history showed c6id.16xlarge at ~$3.08/h in every us-east-1 AZ, essentially on-demand. m6id.16xlarge (same Ice Lake generation, same 64 vCPU, 4x the RAM at 256 GB, 2x1.9 TB NVMe) was ~$0.88-1.10/h spot.
Tips: - Check several families in one loop before launching: aws ec2 describe-spot-price-history --instance-types $t --product-descriptions Linux/UNIX --start-time $(date -u +%FT%TZ) --query 'min_by(SpotPriceHistory,&SpotPrice).[AvailabilityZone,SpotPrice]'. - The 'more expensive' general-purpose family can be cheaper on spot than the compute-optimized one, because spot price tracks demand, not list price. - The cheapest AZ may have no capacity (InsufficientInstanceCapacity). The error message lists the AZs that do; retry in the next-cheapest one. - For one-shot jobs use SpotInstanceType=one-time, InstanceInterruptionBehavior=terminate and --instance-initiated-shutdown-behavior terminate. Add a 'shutdown -h +N' in user-data as a hard cost cap, plus an idle watchdog (load < 0.5 for N minutes). Make the watchdog window longer than any expected low-CPU phase such as large network copies.
Follow-up caveat: the cheap m6id.16xlarge spot pool (us-east-1) reclaimed our instance about 40 minutes after boot (StateReason Server.SpotInstanceTermination), and the instance-store NVMe with all intermediate results went with it. For multi-hour benchmark runs on spot: pull the small result files (jsonl/csv/logs) back to durable storage after every stage, not once at the end, and keep each stage resumable. If the run can't tolerate a restart, on-demand may be worth the ~3.7x price.