Scrapy 'twisted.internet.error.ReactorNotRestartable' running spiders one after another; and no, Scrapy is not incompatible with asyncio (2.20 defaults to the asyncio reactor)
Fix for ReactorNotRestartable: register all spiders on one CrawlerProcess and call start() once: p = CrawlerProcess(settings); p.crawl(QuotesSpider); p.crawl(BooksSpider); p.start(). Both run concurrently in the same reactor. A second CrawlerProcess(...).start() in the same Python process raises twisted.internet.error.ReactorNotRestartable, because the Twisted reactor can be started only once. If spiders must run strictly one after another, chain them with CrawlerRunner and yield runner.crawl(...) inside a reactor callback, or use one subprocess per crawl. A common wrong conclusion here is that Scrapy cannot be mixed with asyncio. In Scrapy 2.20 the default TWISTED_REACTOR is 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', callbacks can be async def, and inside them you can await asyncio code, including httpx.AsyncClient requests. Checked: an async def parse() that awaited asyncio.sleep() and an httpx call ran fine, with is_asyncio_reactor_installed() == True, alongside a second spider in the same CrawlerProcess. The real limit is narrower: CrawlerProcess wants to own the reactor and its loop. So you cannot start it twice, and starting it from code that already runs an asyncio loop (a FastAPI handler, a Jupyter cell) likely fails too; use CrawlerRunner or a subprocess there. Verified with Scrapy 2.20.0 and httpx 0.28.1.