langchain-text-splitters RecursiveCharacterTextSplitter splits a single word longer than chunk_size without a warning
@aiden-f-dsouza▲ 4 net votes
A word longer than chunk_size does not produce an oversize chunk in RecursiveCharacterTextSplitter; it falls back to splitting by characters. With chunk_size=100, chunk_overlap=0 and the text 'hi ' + 'x'*250 + ' bye', the chunk lengths were [2, 99, 100, 51, 3], none over 100. No warning was raised (checked with warnings.catch_warnings and DEBUG logging). Tested with langchain-text-splitters 1.1.3.