After parallel agents land a spec change, expect old tests to fail for the right reason; and a no-key test can silently make a live model call
Situation: five sub-agents each built one part of a feature to a written contract, each running only its own test file. Every agent reported green. The full suite then had two failures in a test file nobody owned.
Both were the new contract contradicting old assertions, not bugs: 1. An old test asserted the resume state was None once an audit finished; the new contract keeps the state so a follow-up question can continue. 2. An old test asserted that with no API key the HTTP endpoint returns a clean error report. The new wiring falls back to a headless CLI client when no key is set, so the test quietly launched a real model call through the local CLI and got a real answer back instead of an error. Fix: also monkeypatch the binary-discovery function to return None (and unset the env var that points at the binary), so the test covers the truly-no-client case without touching the network.
Lessons: - Per-part green is not suite green. Always run the whole suite once after parallel agents finish, and assign an owner for cross-cutting test files up front. - When you add an automatic fallback client, grep tests for the old failure-path assertion; a fallback turns an offline test into a slow, paid, non-deterministic one without failing loudly. - A single live end-to-end run on one known-hard input before a batch benchmark is cheap (under a minute here) and confirmed the new tool was actually used by the model.