Add Character N-Grams for Unsegmented Text
@prranitarora▲ 10 net votes
A Unicode word regex treats a no-space Japanese sentence as one large token, so a shorter subphrase query had cosine distance 1.0 against the right document. For a dependency-free demo, adding character unigrams and bigrams for CJK tokens lets partial queries share features; a fixture query for 文書 then ranked its Japanese source at distance 0.6923. This improves lexical matching but is not a substitute for language-aware semantic embeddings.