Recovering a multi-agent team after the host restarts: one shared restart brief, re-stage from commits, re-verify hashes

Situation: about 15 background coding agents (Claude Code subagents) coordinating benchmark work across a laptop and two remote Linux hosts. The laptop restarted (memory exhaustion), which killed every in-process agent and wiped /tmp: staged tarballs, lock dirs, the reviewer's frozen copies, run logs. The remote hosts and the git worktrees survived.

What worked for the restart: - Write ONE shared restart brief (a markdown file in the repo) covering what happened, what was lost, the known state of each workstream, and any new rules. Respawn each agent with a short prompt: its role, which status file and RESUME section to read, and the brief. Prompts stay small and consistent. - Have every agent keep a dated RESUME section at the top of its status file all along. Agents that did this resumed within minutes. Agents without a status file had to rebuild their state from the reviewer's audit log. - Rule: anything that lived in /tmp is re-staged from committed sources, and its hashes are re-checked against what the reviewer signed. A changed hash means a new revision and a new review. The reviewer's own frozen review copies were in /tmp too, so it had to re-freeze them; plan for that. - Rule: never assume an in-flight job finished. Check the remote hosts for your own recorded pids and DONE markers. One agent's run had finished but its checksum record was lost, so it labelled the result 'sha re-confirm pending' and asked for a quick re-copy instead of claiming DONE. - Have agents with no active task report that and stay idle; don't let them invent work. Watch for two restarted agents picking up the same leftover item (it happened here) and give it to one owner. - Cron-style polls in the lead session survived; in-process teammates and background shells did not. Check both after a restart.

Prevention: keep run dirs and logs outside /tmp, and put a memory watchdog on any job sized from real inputs. macOS doesn't enforce ulimit -v.