Multi-agent teams: stuck-work detection must run outside the coordinator and must wake, not just flag
Observed in a large LLM agent team: one lead session and many in-session subagents. The team had a progress auditor agent that logged stale agents (no progress-file write for 30+ min). Work still sat idle for 40-100 min: a signed go-ahead sat unsubmitted, and a gate result went unread.
Why: 1. The auditor flagged but did not wake. Its action was to mark the gap's work unaccepted, a penalty after the fact. 2. The auditor lapsed on its own waits, the same failure it was checking. 3. Everything ran inside the lead's session. When the lead was saturated (human questions, context compaction), all wake-ups waited. 4. The job watcher aged out completions older than 45 min, so misses vanished silently.
What helped: - An independent mechanical checker, in a separate process, every 15 min. It flags any subagent transcript modified in the last 3 h but silent 20+ min, using file mtimes and no LLM. - A second-opinion agent on another machine, given a standing "stuck watch" job. It writes per-item STUCK lines (who must do what) to the lead, and escalates to the human only after 3 consecutive ticks. - No age-out: list overdue items most-overdue-first until acknowledged, and count lapses per hour.
Gotcha: pkill -f "python3 foo.py" inside a shell command also matches the shell running that command line, so it kills itself (exit 144). Use pgrep -f "^python3 foo.py" and kill by PID.