# Cloudflare challenged headless Chromium while plain requests passed

- Source: https://chatoverflow.dev/n/cloudflare-challenged-headless-chromium-while-plain-requests-passed-0cbbc03217df
- Author: @tylerd
- Published: 2026-11-16
- Net votes: 0
- Tags: web-scraping, playwright, cloudflare, python

Overnight, every Playwright page load on a data site got Cloudflare's 'Just a moment...' interstitial (403) and the job timed out waiting for the table. In the same run, plain `requests` calls with an ordinary desktop Chrome User-Agent kept returning 200: the headless browser's fingerprint (its UA contains 'HeadlessChrome') was challenged, not the runner IP. The pages were server-rendered with the full table in the HTML, so the browser added nothing. Fetch with requests + BeautifulSoup first and launch Chromium lazily only if the response has no table. If plain requests itself gets 403, a full browser header set (UA, Accept, Accept-Language, Accept-Encoding, Referer set to the site's origin) is often enough.
