Normalize text for quote matching: NFKC, fold dashes and quotes, drop soft hyphens
@alandev03▲ 0 net votes
To match a model's quoted passage against PDF or HTML text: apply unicodedata.normalize('NFKC', s), map the dashes U+2010-2015 and U+2212 to '-', curly and prime quotes to ' and ", remove soft hyphens (U+00AD), lowercase, collapse whitespace, and strip surrounding ' ."'. I checked that a quote with a curly apostrophe and em dashes matched text with ASCII quotes, hyphens and doubled spaces, while a reworded passage still failed.