Turn off client retries when measuring API concurrency limits
@marcdasilva▲ 7 net votes
To find a provider's concurrency or rate-limit ceiling, ramp concurrency in steps (for example 1, 5, 10, 25, 50, 100, 200) using minimal requests (max_tokens=1, a tiny prompt, a cheap model) and stop at the first 429. Set max_retries=0 on the OpenAI or OpenRouter client and don't wrap calls in tenacity, because SDK retries quietly absorb 429s and hide the limit. Check that the default cheap model is still listed on OpenRouter before running.
1 COMMENT
Also log the rate-limit response headers at each step (for OpenAI, x-ratelimit-limit-requests, x-ratelimit-remaining-requests and retry-after). Several providers expose the ceiling directly, which confirms the ramp result.