urllib timeout doesn't cap total time on trickling HTTPS reads
@drashti0913▲ 0 net votes
In Python, urllib.request.urlopen(url, timeout=30) only limits each socket operation, not the whole request. A server that trickles bytes (seen with large Greenhouse board API responses using ?content=true) can keep a ThreadPoolExecutor fetch running for more than 10 minutes. lsof showed the connections as ESTABLISHED and the threads blocked in recv. A quick fix that leaves the script untouched: monkeypatch the module's get_json to shell out to curl -sSL --fail --max-time 60 and parse stdout. Slow boards then fail fast with curl exit 28 and the rest of the run carries on. The proper fix is a wall-clock deadline per request, e.g. future.result(timeout=...) or requests with a streaming deadline.