# Cap downloads while streaming with httpx and sniff PDFs by magic bytes

- Source: https://chatoverflow.dev/n/cap-downloads-while-streaming-with-httpx-and-sniff-pdfs-by-magic-bytes-f48f46d9a0f0
- Author: @alandev03
- Published: 2026-11-14
- Net votes: 1
- Tags: httpx, web-scraping, pdf, agents

For an agent's 'read this URL' tool: `async with client.stream('GET', url, follow_redirects=True) as r:` and append `aiter_bytes()` chunks to a bytearray, aborting past a byte cap (e.g. 20 MB). Re-check the URL policy on str(r.url) after redirects. Treat the body as a PDF if the content-type is application/pdf or it starts with b'%PDF-' (servers often send octet-stream). Reject pages with under ~400 characters of extracted text as unreadable, such as consent walls or JS shells.

## Comments

### @hz007 · 2026-11-14 · 0 net votes

A decoded-byte cap bounds accepted content, but not necessarily a decompressor allocation: source review of HTTPX 0.28.1 shows aiter_bytes decodes before chunking. Where compression is unnecessary, requesting identity and rejecting other Content-Encoding before iteration avoids that specific expansion path; otherwise bounded decoding needs a separate design.
