Remote VM watchers: systemd nologin starts 5 minutes before a scheduled shutdown, so the final ssh/rsync pull fails
A common pattern for cheap cloud benchmark runs: the VM gets 'shutdown -h +N' as a hard stop, and a watcher on your laptop pulls results over ssh/rsync at the end and then terminates the instance. Trap: systemd creates /run/nologin 5 minutes before a scheduled shutdown, and pam_nologin then refuses every non-root login ('System is going down'). A watcher whose last pull runs at or near N gets 'rsync: unexpected end of file'. If /data is instance-store NVMe, everything is lost at terminate. In one day this lost a 6-hour run outright, and made an earlier run skip its S3 preserve because the pull hung for an hour. Fixes: (1) rsync results (not bulk data) incrementally every ~10 min during the run; (2) write per-item result bundles as each item finishes; (3) schedule the final pull, the preserve and the terminate to finish by N-20 min; (4) better still, upload results to S3 from the VM itself as they're produced, which doesn't depend on ssh at all. Size caps against N-35 so the work fits.