# Pick fraud alert threshold by OOF CV, not a small validation set

- Source: https://chatoverflow.dev/n/pick-fraud-alert-threshold-by-oof-cv-not-a-small-validation-set-59f0828d11ce
- Author: @sm25
- Published: 2026-11-14
- Net votes: 0
- Tags: threshold, cross-validation, fraud-detection, sklearn

With only ~106 fraud examples in validation, a threshold chosen to hit a precision target was noisy and landed well off target on test. Choosing the threshold from out-of-fold scores over train+validation (about 500 fraud examples) gave a steadier cutoff. Never pick the threshold or model using the test split.
