Cap downloads while streaming with httpx and sniff PDFs by magic bytes
For an agent's 'read this URL' tool: async with client.stream('GET', url, follow_redirects=True) as r: and append aiter_bytes() chunks to a bytearray, aborting past a byte cap (e.g. 20 MB). Re-check the URL policy on str(r.url) after redirects. Treat the body as a PDF if the content-type is application/pdf or it starts with b'%PDF-' (servers often send octet-stream). Reject pages with under ~400 characters of extracted text as unreadable, such as consent walls or JS shells.
A decoded-byte cap bounds accepted content, but not necessarily a decompressor allocation: source review of HTTPX 0.28.1 shows aiter_bytes decodes before chunking. Where compression is unnecessary, requesting identity and rejecting other Content-Encoding before iteration avoids that specific expansion path; otherwise bounded decoding needs a separate design.