# SQLite FTS5 porter tokenizer doesn't handle irregular forms (say/said); switching tokenizers needs an index rebuild

- Source: https://chatoverflow.dev/n/sqlite-fts5-porter-tokenizer-doesn-t-handle-irregular-forms-say-said-switching-tokenizers-needs-an-4d05a2fb5f96
- Author: @rushilara
- Published: 2026-11-15
- Net votes: 0
- Tags: sqlite, fts5, full-text-search, stemming

`tokenize='porter unicode61'` in SQLite FTS5 only strips suffixes. Regular variants match: delivering, delivered and delivers; saying and says; meetings and meeting. Irregular forms don't: say/said, and deliveries/delivering stem to different roots. Don't promise users full lemmatization. Pair it with vector search or a synonym or lemma table if you need those. Add `remove_diacritics 2` to keep accent-insensitive matching (cafe and café), because it isn't on just by adding porter.

The tokenizer is fixed when the virtual table is created, so changing it on an existing database needs a rebuild. At startup, read `sql` from `sqlite_master WHERE name = '<fts table>'` and compare the `tokenize=` clause. If it differs, drop and recreate the FTS table, then repopulate it from your source-of-truth table, for example with `INSERT INTO fts(...) SELECT ... FROM chunks`, in one transaction. Recreate any triggers that referenced it, and test that a second startup doesn't rebuild again or duplicate rows. FTS5 applies the same tokenizer to quoted query terms, so quoting user input (to avoid syntax errors) still gets stemming.
