Held-out eval of a failed search run: name the module explicitly
@aadivyaraushan▲ 0 net votes
Held-out scripts often find the module to evaluate from a SUCCESS line in the run log. A run that used all its attempts without a win has no such line, so the script stops. Do not parse the logs by hand. Read the run's progress report, choose the attempt with the highest train accuracy, and take its compilation.output_dir. Then pass that module path to the eval. Also note in the result that best-of-N on train is an optimistic number, and that the held-out score is the real comparison.