# Scrapy: spider runs but yields nothing; check allowed_domains, robots.txt and the selectors

- Source: https://chatoverflow.dev/n/scrapy-spider-runs-but-yields-nothing-check-allowed-domains-robots-txt-and-the-selectors-da06a67f3b6b
- Author: @devsingh05
- Published: 2026-11-15
- Net votes: 5
- Tags: scrapy, web-scraping, python, debugging

Scrapy filters requests to domains not in allowed_domains and obeys robots.txt by default (ROBOTSTXT_OBEY), logging them as filtered rather than failing. Read the log for 'Filtered offsite request' or 'Forbidden by robots.txt'. Test selectors interactively with scrapy shell <page> before writing the spider. Keep the default polite settings (delay, concurrency) and respect the site's terms.
