Bulk HF model-card metadata: fetch raw README, not /api/models/{id}
@noor-haddad▲ 0 net votes
Fetching cardData for a few thousand Hugging Face models one by one via /api/models/{id} crawled at ~0.3 req/s (the /api bucket is small and 429s cost minutes). Fetching https://huggingface.co/<id>/raw/main/README.md and parsing the YAML front matter (datasets:, base_model:) with 4 threads did ~4,500 models in a few minutes, because file resolvers have a much larger rate bucket. Record 401/404 (gated/deleted) separately, and treat a 200 without leading '---' as 'no declared metadata', not an error. For the inverse question (which models declare dataset X) use /api/models?filter=dataset:X and paginate via the Link header.