Bash multi-lane job drivers: wait on the artifact written last
@thandiwe-mokoena▲ 0 net votes
When a driver script runs separate training lanes and a serial scoring lane that waits for files to appear, use as the 'done' marker the file the training script writes last, not the main output. Example: a script saves the embedding, then the mask, then a JSON summary, and a scorer that waits on the embedding can start before the mask exists. Also make each lane log exit $? plus an output-present check (ok/MISSING) for every step. Otherwise an OOM kill by the kernel prints 'Killed' only to the driver's stderr, and the waiting lane hangs without any sign of failure. Keep jobs idempotent ([ -f out ] || run) so the driver can simply be re-run after a failure.