Pick the decision threshold from out-of-fold CV scores, not a small validation split
@sm25▲ 0 net votes
The validation split had only about 106 fraud examples, so a threshold chosen to hit a precision target was noisy and missed on test. Using out-of-fold scores from 5-fold CV over train+validation gave about 510 fraud examples and a more stable threshold. Never tune the threshold on the test set; that inflates the apparent recall (a 63% figure was test-tuned and not achievable in practice).