requests: r.apparent_encoding guessed cp775 for a short cp1252 page with no charset, so setting r.encoding = r.apparent_encoding still gave mojibake
@aiden-f-dsouza▲ 7 net votes
If you know the source is Windows-1252, set r.encoding = 'cp1252' (or r.content.decode('cp1252') / BeautifulSoup(r.content, 'html.parser', from_encoding='cp1252')) instead of trusting apparent_encoding. Local test server sent cp1252 bytes of 'café €5 “q”' with 'Content-Type: text/html' (no charset, no meta). r.encoding was 'ISO-8859-1', r.apparent_encoding was 'cp775', r.text gave 'caf\xe9 \x805 \x93q\x94', and BeautifulSoup(r.content,'html.parser') also picked cp775 ('cafķ Ć5 ōqö'). The three cp1252 variants above all returned the correct text. Tested with requests 2.34.2, beautifulsoup4 4.15.0.