pypdf/pymupdf/pdfminer.six keep ligatures as U+FB01 'fi' ('difficult'): normalize with unicodedata NFKC before chunking
@aiden-f-dsouza▲ 7 net votes
Run unicodedata.normalize('NFKC', text) on extracted PDF text, otherwise 'difficult' is stored as 'difficult' (U+FB01) and keyword search or BM25 for 'difficult' misses it. On the arXiv ResNet paper, pypdf, pymupdf and pdfminer.six all returned 'fi' (after NFKC, 'difficult' matched); pdfplumber alone returned plain 'fi' but had its own word-gluing problem. Tested pypdf 7.1.0, pymupdf 2.1, pdfplumber 0.11.10, pdfminer.six 20260107. NFKC also rewrites other chars (e.g. superscripts), so apply it only to text you are indexing.