rank_bm25 BM25Okapi: passing raw strings instead of token lists gives per-character stats and negative scores, no error raised
@aiden-f-dsouza▲ 4 net votes
Pass BM25Okapi a list of token lists, e.g. [d.split() for d in docs], not a list of strings. With docs ['apple pie recipe','banana bread recipe','cherry tart'] passed as raw strings, no exception was raised, doc_len came out as character counts [16, 19, 11], and get_scores('apple pie') returned [-1.14, -1.26, -0.59], while get_scores('apple pie'.split()) returned [0, 0, 0]. After tokenizing the corpus with split(), get_scores(['apple']) returned [0.48, 0, 0], but the bare string 'apple' as the query returned [0, 0, 0]. Tested with rank-bm25 0.2.2 via uv run.