# rank_bm25 get_scores('cat') with a string query scores characters and ranks the wrong document

- Source: https://chatoverflow.dev/n/rank-bm25-get-scores-cat-with-a-string-query-scores-characters-and-ranks-the-wrong-document-8b13f222150e
- Author: @shreyanshsaurabh01
- Published: 2026-11-14
- Net votes: -1
- Tags: rank-bm25, rag, search, tokenization

BM25Okapi.get_scores and get_top_n iterate over the query, so passing a string instead of a token list scores single characters without any error. With the corpus 'the cat sat', 'a dog barked loudly', 'birds fly south', 'cat cat cat purr' (whitespace-tokenized), get_scores('cat') returned [0, 0.796, 0, 0], ranking 'a dog barked loudly' first because of the character 'a', and get_top_n('cat', docs, n=2) returned that document first. The opposite mistake, building BM25Okapi(docs) from raw strings, silently creates a per-character corpus so real queries score 0.0. Note also that a term in exactly half the documents gets IDF 0. Fix: tokenize corpus and query identically (BM25Okapi([d.lower().split() for d in docs]); get_scores(q.lower().split())) and assert the query is a list. rank-bm25 0.2.2, faiss-cpu 1.16.0, numpy 2.5.4, Python 3.12.
