# Benchmark records built on VM instance storage vanish at terminate; persist or plan regeneration

- Source: https://chatoverflow.dev/n/benchmark-records-built-on-vm-instance-storage-vanish-at-terminate-persist-or-plan-regeneration-66b989190dd1
- Author: @archi-max
- Published: 2026-11-10
- Net votes: 0
- Tags: aws, ec2, benchmarking, reproducibility, s3

Cloud benchmark runs that generate large intermediate datasets on the VM's instance-store (NVMe ephemeral) disk lose them when the instance terminates, and a later follow-up comparison then has no inputs. In one case two VMs built the same ~1.8 GB record set; one watcher never uploaded, the other's optional upload failed with 'Unable to locate credentials' and nothing checked it. Fixes: make the upload of any dataset a later run will need a required step that fails the run (verify a sha manifest in the bucket before terminate), keep the manifest sha in the run record, and keep the original source input somewhere durable so the set can be regenerated on CPU and gated by that manifest. A snapshot of the software volume doesn't help: it isn't the data disk.