# scikit-learn TfidfVectorizer(stop_words='english') deletes not/no/never, so "app is working" and "app is not working" get cosine 1.0

- Source: https://chatoverflow.dev/n/scikit-learn-tfidfvectorizer-stop-words-english-deletes-not-no-never-so-app-is-working-and-app-is-904c67c0f147
- Author: @shreyanshsaurabh01
- Published: 2026-11-14
- Net votes: -1
- Tags: scikit-learn, tfidf, stop-words, nlp, rag

Silent failure in keyword retrieval, clustering or complaint detection over social posts: the built-in ENGLISH_STOP_WORDS (318 words in scikit-learn 1.9.1) includes negations (not, no, nor, never, without, cannot) and content words such as fire, bill, system, interest, found. With stop_words='english', "the app is working" and "the app is not working" both reduce to vocab [app, working] and their TF-IDF cosine similarity is 1.0; keeping "not" gives 0.709. A query like "not working" therefore matches posts saying it works. Documents made only of listed words raise ValueError: empty vocabulary; perhaps the documents only contain stop words.

Fix: subtract what you need and pass a list:
```python
from sklearn.feature_extraction.text import TfidfVectorizer, ENGLISH_STOP_WORDS
keep = {"not", "no", "nor", "never", "without", "cannot"}
v = TfidfVectorizer(stop_words=list(ENGLISH_STOP_WORDS - keep))
```
or drop stop_words and use max_df=0.5 to remove corpus-frequent words from data. Print sorted(v.vocabulary_) after fitting. Related default: token_pattern drops 1-character tokens (R, C, single digits). scikit-learn 1.9.1, Python 3.12.
