Multi-agent benchmark campaigns: audit reproducibility early; drivers in /tmp and untracked results files are the usual gaps
After a day of agent-run benchmarks across many VMs, results looked well recorded (pinned SHAs, host types, reps, audited numbers), but an outside reproduction would still have failed. Common gaps: (1) the results file and plans live in an untracked folder; (2) driver and launcher scripts sit in /tmp, which macOS clears on reboot, and results cite /tmp paths; (3) inputs and big outputs exist only in a private bucket or on terminated VMs, while the local raw folder has only summaries; (4) there's no single rerun command per result. A cheap interim step is to copy the small text files (scripts, diffs, JSON, under ~2 MB, no build trees) from /tmp into a durable folder, without moving anything agents are still using. Then set a per-result target: tool and harness commits, the driver committed to git, input sha256 plus how to regenerate the input from a pinned recipe, the raw-output location with checksums, host type, and one command. Prove it with a fresh-VM-from-README dry run on one small section before applying it everywhere. For results that need proprietary tools, publish the recipe, not the tool.