rank_bm25 with query.lower().split() misses API paths: 'POST /accounts/{id}/transfers?' stays one token and ranks the wrong endpoint; tokenize with re.findall(r'\w+')
Tokenize documents and queries for BM25 with re.findall(r'\w+', text.lower()), not text.lower().split(). Whitespace splitting keeps punctuation glued on, so in a corpus of API docs (OpenAPI spec chunks such as Nessie v2's) a path like /accounts/{id}/transfers and a question ending in ? become tokens that never match anything. BM25 then ranks on filler words.
Reproduced with rank-bm25 0.2.2 on five short endpoint chunks, query How do I POST /accounts/{id}/transfers?:
- split: tokens ['how','do','i','post','/accounts/{id}/transfers?']; the transfers chunk scored 0.33 and the *purchases* chunk 0.46 (wrong endpoint first; only 'post' matched)
- regex: tokens ['how','do','i','post','accounts','id','transfers']; transfers 1.89, purchases 1.06, GET /accounts/{id} 0.60 (right order)
Use the same tokenizer for indexing and querying (BM25Okapi([tok(d) for d in docs]), bm25.get_scores(tok(q))); a mismatch is the other common way to get all-zero scores. \w+ keeps snake_case intact (balance_cents stays one token, which is usually what you want for field names). If you need {id} vs id or camelCase splits, add a custom regex, but the plain \w+ version was enough to lift a hybrid BM25+vector retriever over the Nessie docs from below vector-only to +12 pp Hit@5, together with min-max score normalization.