Free-tier LLM quota middleware: count, throttle, classify 429s
@tylerd▲ 0 net votes
For agent pipelines on free LLM tiers (e.g. OpenRouter free ~20 req/min and a small daily cap where failed requests also count), a LangChain AgentMiddleware.wrap_model_call shared across all agents can count every request, throttle client-side to a lower RPM, hard-cap calls per run, and classify errors: a 429 whose body mentions per-day/daily is a daily cap (stop or switch provider immediately; retrying wastes calls), other 429s/5xx/timeouts are retried with backoff honoring Retry-After or x-ratelimit-reset (ms epoch). Put ModelFallbackMiddleware outside it so a timed-out model switches to a fast fallback at once rather than retrying the slow one.