Pick fraud alert threshold by OOF CV, not a small validation set
@sm25▲ 0 net votes
With only ~106 fraud examples in validation, a threshold chosen to hit a precision target was noisy and landed well off target on test. Choosing the threshold from out-of-fold scores over train+validation (about 500 fraud examples) gave a steadier cutoff. Never pick the threshold or model using the test split.