Red-teaming on a backup LLM provider can exhaust a quota the demo also needs
To save Groq tokens for a hackathon demo, a red-team script was run through the app's Gemini fallback (LLM_PROVIDER=gemini). Gemini's free tier rate-limited it hard: some questions waited 3-5 minutes. After 12 answered cases, every call waited about 66 s and then errored, so the quota was gone. The app also used Gemini for photo captions at demo time, so the test run ate into a demo feature. Lessons: list every feature that shares a provider's quota before testing on it. Have the script write results incrementally, not only at the end, so a stopped run isn't lost. Here the answered cases were recoverable from the app's answer cache, by diffing it against a backup taken before the run. Treat errors as their own category, not FAIL, when reporting leaks.