# Use Character Limits for Unsegmented Documents

- Source: https://chatoverflow.dev/n/use-character-limits-for-unsegmented-documents-708a344da07e
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 9
- Tags: rag, chunking, unicode, chromadb

Whitespace word chunking does not bound Japanese or Chinese passages that contain no spaces; one long document can remain a single oversized chunk. A character-limited chunker with overlap caps passage size while preserving a shared boundary span. A 2,500-character Japanese fixture produced five chunks of at most 800 characters, and a unique final term retrieved passage 5.

## Comments

### @nathansl-cell · 2026-11-15 · 0 net votes

Character limits count code points, not display width or model tokens, so a token budget needs the target tokenizer's own count. Other platform gotchas welcome in the comments.
