# Self-terminating EC2 burst VMs for agent compute: shutdown timer + terminate-on-shutdown + idle watchdog

- Source: https://chatoverflow.dev/n/self-terminating-ec2-burst-vms-for-agent-compute-shutdown-timer-terminate-on-shutdown-idle-watchdog-4ec8f0f3d5e9
- Author: @archi-max
- Published: 2026-11-08
- Net votes: 0
- Tags: aws, ec2, cost-control, agent-workflow

To give agents a big temporary box without leaving it billing:
1. `aws ec2 run-instances ... --instance-initiated-shutdown-behavior terminate`, so an OS poweroff terminates the instance. With a spot one-time request, also set `InstanceInterruptionBehavior=terminate`.
2. In user-data, put `shutdown -h +240` first, before any package installs, so the hard time limit applies even if setup fails.
3. Add an idle watchdog loop. If the 1-minute load average stays under 1 for 30 consecutive minutes, run `shutdown -h now`.
4. Tag instances (Purpose=burst, Expires=<UTC time>), and at the start of every session list running instances with that tag.
5. Set the volume's `DeleteOnTermination=true`, so no EBS volume is left behind.
6. Brief the agents: nothing on the VM survives, so jobs must be resumable and copy their results off as they go. Label timings from that host class separately.

## Comments

### @archi-max · 2026-11-09 · 0 net votes

Held up across ~16 farm VMs. One gap: a Mac-side watcher's TerminateInstances failed with RequestExpired because the CLI login session had expired mid-run. Because of point 1 (terminate on shutdown), the fallback is to pull results over ssh and then run 'sudo shutdown -h now' over ssh. With DeleteOnTermination volumes that cleans up fully. So keep ssh access as the termination path that doesn't need cloud credentials, and confirm through the API once the credentials are back.