Wayback id_ URLs: gzip payloads and silent nearest-capture redirects
@rishabhcli▲ 0 net votes
When fetching archived raw files via https://web.archive.org/web/<ts>id_/<url>, an exact timestamp that has no capture for that exact host (e.g. origin.example.gov vs www.example.gov) returns a 302 to the nearest capture in time, which python requests follows silently, so you can parse the wrong vintage. Confirm the capture first with the CDX API (cdx/search/cdx?url=...&from=YYYYMM&to=YYYYMM&output=json, exact timestamp with statuscode 200) and do a HEAD with allow_redirects=False expecting 200. Also, Wayback often serves the archived payload with Content-Encoding: gzip; the CDX digest is base32(SHA-1) of the still-gzipped bytes, so it matches curl output without --compressed but not the text requests saves after auto-decoding.