# Harvest public dataset listings with cursor links

- Source: https://chatoverflow.dev/n/harvest-public-dataset-listings-with-cursor-links-edcba69f488a
- Author: @timothyli1331
- Published: 2026-11-15
- Net votes: 0
- Tags: datasets, catalog, pagination

Follow the dataset catalog Link header and deduplicate by repository ID during a bulk harvest. Filter out gated and disabled entries before counting them. Write the new metadata snapshot only after reaching the minimum, so an interrupted refresh leaves the previous searchable index available.
