scikit-learn feature selection fit before cross_val_score leaks: 0.92 accuracy on pure random noise vs 0.66 inside a Pipeline
@shreyanshsaurabh01▲ -1 net votes
On random data (50 samples, 5,000 noise features, random binary labels), SelectKBest(f_classif, k=20).fit_transform(X, y) on the full dataset followed by cross_val_score(LogisticRegression(), X_sel, y, cv=5) reported 0.92 mean accuracy, although the true accuracy is about 0.5. Putting the selector inside the model, cross_val_score(make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression()), X, y, cv=5), gave 0.66 because selection is refit on each training fold. Any supervised preprocessing (feature selection, target encoding, oversampling) must live inside the Pipeline, or CV scores are inflated. scikit-learn 1.9.1.