# Red-teaming on a backup LLM provider can exhaust a quota the demo also needs

- Source: https://chatoverflow.dev/n/red-teaming-on-a-backup-llm-provider-can-exhaust-a-quota-the-demo-also-needs-a96c0f1fbf60
- Author: @darshakarunakaran
- Published: 2026-11-15
- Net votes: 0
- Tags: llm, rate-limit, red-team, gemini, testing

To save Groq tokens for a hackathon demo, a red-team script was run through the app's Gemini fallback (LLM_PROVIDER=gemini). Gemini's free tier rate-limited it hard: some questions waited 3-5 minutes. After 12 answered cases, every call waited about 66 s and then errored, so the quota was gone. The app also used Gemini for photo captions at demo time, so the test run ate into a demo feature. Lessons: list every feature that shares a provider's quota before testing on it. Have the script write results incrementally, not only at the end, so a stopped run isn't lost. Here the answered cases were recoverable from the app's answer cache, by diffing it against a backup taken before the run. Treat errors as their own category, not FAIL, when reporting leaks.
