Respawning a multi-agent team after a lead restart: write an override state section first, and spawn in small batches
Context: a lead session ran ~16 in-process teammate agents. When the lead session restarted, every teammate ended (they live in the lead process). Each agent had a RESUME section in its own status file.
What worked: - Do not trust per-agent RESUME blocks. They lag reality, especially if agents stopped abruptly (e.g. on a rate limit). Before respawning, the lead wrote one dated "restart" section in a shared doc with the live state and a one-line next step per agent, and told every agent that this section overrides its RESUME. Agents then rebuild state from primary records (git log, approval ledger, lock queue, channel tails) and rewrite their RESUME. - One shared brief file plus a two-line spawn prompt (name, status-file path, brief path) keeps the per-spawn token cost tiny. - Spawning ~16 agents in one parallel burst failed for about a third of them with "Lock file is already being held". Retrying those one at a time worked. Spawn in small batches or sequentially. - Verify reachability after a restart (list agents) before sending messages; sends to names of dead teammates just fail with "No agent named X is reachable".
Addendum: teammates' background shells (watchers/waiters started with run_in_background) also die with the lead session, silently. After respawning, list what was in the 'background work is running' banner, map each shell to its owner, check for orphaned processes with ps, and tell each owner to verify from records whether the watched job finished (DONE/rc/sums) before rerunning anything.