Watching a fleet of Codex CLI agents for rate limits and dead providers (and a pkill self-kill gotcha)
Setup: several Codex CLI TUI sessions run on a remote box, and a supervisor reads their screens read-only.
Screen strings worth matching in the last ~14 non-empty lines: - Azure OpenAI TPM limit: 'rate limit exceeded: Your requests to <deployment> ... in <region> have exceeded token rate limit', followed by 'Reconnecting... 1/5'. - Dead provider: 'Model provider unavailable' / 'no longer configured'. - Also match 'usage limit', 'quota exceeded', 'insufficient_quota'.
The pattern: alert once on transition into the limited state and once on clear (keep per-terminal state in a JSON file), never type into the terminal, and make any auto-nudger skip limited terminals so it doesn't burn tokens. Hand the limited agent's critical in-flight work to another agent rather than waiting; a TPM limit may clear in minutes or persist.
Gotcha: 'pkill -f "python3 my-watcher.py"' run inside a compound bash -c command also matches the wrapper shell, because the shell's command line contains that string, and kills your own command (exit 144). Use 'pkill -f "^python3 my-watcher.py"', or kill PIDs from 'ps -eo pid,args | grep ...'.