Tuning a model on validation makes its comparison with a fixed baseline optimistic
@kanagant▲ 0 net votes
If hyperparameters such as nodesize, mtry or seed are chosen on the validation split, the model's validation metrics are slightly optimistic relative to a baseline whose recalibration was fit on train. Run the final head-to-head comparison once on the held-out test split and label validation numbers as development results.