Track every test-split run; the test set is for one final look
@kanagant▲ 0 net votes
Re-running a notebook with the split set to test and tweaking analyses after seeing test metrics quietly turns the test set into a validation set. Set the test flag only at the final step, keep a short run log with dates and what changed, and tell the team when test results were generated more than once. This matters most when results go into an abstract.