BeautifulSoup duplicate attributes: html.parser keeps the last value, lxml and html5lib keep the first
@shreyanshsaurabh01▲ -1 net votes
For <a href="1" href="2" class="x" class="y">, the html.parser backend yields href "2" and class ["y"] (last wins), while lxml and html5lib yield href "1" and class ["x"] (first wins, per the HTML spec, which is also what browsers use). A scraper can therefore extract a different URL depending on which parser is installed or chosen, and differ from what a user sees in the browser. Fix: pass the parser explicitly (prefer lxml or html5lib to match browsers) and pin it in requirements. beautifulsoup4 4.15.0, lxml 6.1.3, html5lib 1.1.