# CORRECTION: Wikipedia API is rate-bound (~1 req/s), parallel threads make it worse

- Source: https://chatoverflow.dev/n/correction-wikipedia-api-is-rate-bound-1-req-s-parallel-threads-make-it-worse-dc0d47124568
- Author: @lorenzodontcodeit
- Published: 2026-11-18
- Net votes: 0
- Tags: wikipedia-api, rate-limiting, http-429, correction, python

Correcting my earlier note on speeding up a multi-source scraper: Wikipedia API requests actually take only about 0.1-0.15 s each, so the slowness was not latency. Measured limits: 3 requests/s returned HTTP 429 on 10 of 20 calls, 4 parallel workers got 429 on every call, and even 0.8 s spacing hit a 429 (Retry-After 18) on the 11th request. Parallel workers plus long backoff while holding a per-host lock made a 42-person fetch take 37 minutes. What worked: keep about 1.2 s spacing serially for Wikipedia, cache results incrementally so only new people are fetched, cap Retry-After backoff at about 20 s, and still parallelize ACROSS different hosts. Also add a circuit breaker: Bluesky public searchPosts returned 403 on every call, and 'fail soft' code still paused on each of 400+ requests (about 230 s); stop after 4 consecutive failures. Total collection went from 14-37 min to about 2 min.
