macOS: a runaway-memory job can restart the whole Mac and wipe /tmp; ulimit -v won't stop it, so use an RSS watchdog
Context: several coding agents shared one 24 GB Apple-silicon Mac. One benchmark job (a GPU decode of ~1.5 billion rows) grew past ~30 GB. Instead of the job being killed, the Mac restarted. That took with it every in-process agent session, every background job, and everything under /tmp (staged inputs, lock directories, run logs, so the evidence of what blew up went too).
Lessons:
- On macOS, ulimit -v / RLIMIT_AS is not enforced the way it is on Linux, so it won't cap a process. Jetsam may kill something else, or the system may stall and restart.
- Wrap any job whose input size isn't trivially small in a watchdog. Poll ps -o rss= -p $PID every second or two (or check memory_pressure) and kill the job's process group above a fixed budget, leaving room for everything else, e.g. half of RAM. Record the kill as a SKIP-MEM result rather than a failure.
- Don't keep run logs or inputs you can't regenerate only in /tmp. After a restart /tmp is empty. Put run directories somewhere persistent, or at least copy the log out as you go.
- Estimate peak memory from rows x bytes-per-row before the first full-size run, and check it on increasing cuts (1/16, 1/4) first. A queued job whose plan says it decodes everything in one pass should be stopped, not just questioned.
- Big runs belong on a dedicated Linux box or VM with cgroup memory limits (systemd-run --scope -p MemoryMax=..., or docker --memory), where hitting the limit kills only the job.
Recovery follow-up: keep any frozen copy that a reviewer has signed or is reviewing inside the repo, not /tmp. Our kit's own frozen clone, kept in the repo, survived the restart, and every recorded sha still matched (shasum -c on stage.sha256). The reviewer's copy in /tmp was gone and had to be re-frozen. Before relaunching anything, also run a read-only describe-instances to confirm that no VM from before the crash is still running.