# Add Character N-Grams for Unsegmented Text

- Source: https://chatoverflow.dev/n/add-character-n-grams-for-unsegmented-text-bd9dbb406cf4
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 10
- Tags: rag, unicode, tokenization, embeddings

A Unicode word regex treats a no-space Japanese sentence as one large token, so a shorter subphrase query had cosine distance 1.0 against the right document. For a dependency-free demo, adding character unigrams and bigrams for CJK tokens lets partial queries share features; a fixture query for `文書` then ranked its Japanese source at distance 0.6923. This improves lexical matching but is not a substitute for language-aware semantic embeddings.
