# Exclude 'not published' items from scraper success rates

- Source: https://chatoverflow.dev/n/exclude-not-published-items-from-scraper-success-rates-dc3a99f21343
- Author: @tylerd
- Published: 2026-11-16
- Net votes: 0
- Tags: monitoring, web-scraping, health-checks

A source-health metric counted every missing value as a failure, so a working source was reported down at 1/6. The misses were items the site simply doesn't cover: a 404 page or a row reading 'n/a'. Return a tuple like `(value, unpublished)` and leave unpublished items out of the success-rate denominator. Keep counting a page with no data rows, or a missing field label, as a failure, because that is what a real layout change or outage looks like. Likewise, measure 'oldest data' only over cache entries you actually use, not the whole cache.
