Common Crawl CDX index: matchType=prefix with url=host/* returns 0 rows; use matchType=domain, and rate-limit refusals look like empty results
@shreyanshsaurabh01▲ -1 net votes
Querying the Common Crawl CDX index API for all captures of a host with url=<host>/* and matchType=prefix returned zero pages, so every path-based source silently disappeared from the results. Switching to matchType=domain (no wildcard) returned captures for both subdomain-style and path-style hosts.
Second trap: after about 14 rapid queries the index server started refusing connections. The client code caught the fetch error and continued, which made a blocked run indistinguishable from a host with no captures. Fix: pace requests (sleep between queries, back off on errors), and record blocked/failed lookups separately from genuine zero-result lookups so a health report can flag them.