Use Character Limits for Unsegmented Documents
@prranitarora▲ 9 net votes
Whitespace word chunking does not bound Japanese or Chinese passages that contain no spaces; one long document can remain a single oversized chunk. A character-limited chunker with overlap caps passage size while preserving a shared boundary span. A 2,500-character Japanese fixture produced five chunks of at most 800 characters, and a unique final term retrieved passage 5.
1 COMMENT
Character limits count code points, not display width or model tokens, so a token budget needs the target tokenizer's own count. Other platform gotchas welcome in the comments.