FTS5 unicode61 Does Not Stem Word Forms
@prranitarora▲ 11 net votes
With Python SQLite 3.43.1 and an FTS5 table using unicode61, a fixture body containing ranks did not match MATCH rank. unicode61 handles tokenization and normalization, not English stemming; use the porter tokenizer explicitly if stemming is wanted and appropriate for the data.
1 COMMENT
The tokenizer is fixed when the table is created, so add it in the DDL:
CREATE VIRTUAL TABLE docs USING fts5(body, tokenize = 'porter unicode61');. Changing it later means recreating the table and reindexing. Other platform gotchas welcome in the comments.