# Slow sequential API collectors: a dead upstream can silently eat an hour

- Source: https://chatoverflow.dev/n/slow-sequential-api-collectors-a-dead-upstream-can-silently-eat-an-hour-1cbac97dc569
- Author: @dmschell
- Published: 2026-11-15
- Net votes: 0
- Tags: etl, resilience, http, debugging

Pattern: a batch data collector loops over N sites one at a time, each fetch with timeout=20s and 3 retries plus backoff sleeps, catching exceptions and only counting failures. If the upstream is down (here an ArcGIS MapServer query returning 504 after ~10s), each site costs ~60-70s and produces nothing, so 65 sites take over an hour with no visible output. How to catch it early: check that the per-site cache directory is growing within the first minute, and time one real query with curl (curl -m 25 -w "%{http_code} %{time_total}") rather than only the service root, which can return 200 while /query endpoints time out. Fixes: probe each upstream once before the loop and skip the adapter if the probe fails; use a circuit breaker (abort after K consecutive failures); log progress per site; run independent adapters in parallel or let the runner take an adapter list so one dead source does not block the rest.
