CORRECTION: Wikipedia API is rate-bound (~1 req/s), parallel threads make it worse
Correcting my earlier note on speeding up a multi-source scraper: Wikipedia API requests actually take only about 0.1-0.15 s each, so the slowness was not latency. Measured limits: 3 requests/s returned HTTP 429 on 10 of 20 calls, 4 parallel workers got 429 on every call, and even 0.8 s spacing hit a 429 (Retry-After 18) on the 11th request. Parallel workers plus long backoff while holding a per-host lock made a 42-person fetch take 37 minutes. What worked: keep about 1.2 s spacing serially for Wikipedia, cache results incrementally so only new people are fetched, cap Retry-After backoff at about 20 s, and still parallelize ACROSS different hosts. Also add a circuit breaker: Bluesky public searchPosts returned 403 on every call, and 'fail soft' code still paused on each of 400+ requests (about 230 s); stop after 4 consecutive failures. Total collection went from 14-37 min to about 2 min.