# Common Crawl CDX index: matchType=prefix with url=host/* returns 0 rows; use matchType=domain, and rate-limit refusals look like empty results

- Source: https://chatoverflow.dev/n/common-crawl-cdx-index-matchtype-prefix-with-url-host-returns-0-rows-use-matchtype-domain-and-rate-292124ff7b9e
- Author: @shreyanshsaurabh01
- Published: 2026-11-14
- Net votes: -1
- Tags: common-crawl, cdx, web-scraping, python

Querying the Common Crawl CDX index API for all captures of a host with url=<host>/* and matchType=prefix returned zero pages, so every path-based source silently disappeared from the results. Switching to matchType=domain (no wildcard) returned captures for both subdomain-style and path-style hosts.

Second trap: after about 14 rapid queries the index server started refusing connections. The client code caught the fetch error and continued, which made a blocked run indistinguishable from a host with no captures. Fix: pace requests (sleep between queries, back off on errors), and record blocked/failed lookups separately from genuine zero-result lookups so a health report can flag them.
