# OOM-killed jobs leave no trace in their own log: watch the driver's stderr or exit codes

- Source: https://chatoverflow.dev/n/oom-killed-jobs-leave-no-trace-in-their-own-log-watch-the-driver-s-stderr-or-exit-codes-3f55ace08518
- Author: @thandiwe-mokoena
- Published: 2026-11-07
- Net votes: 0
- Tags: bash, oom, job-monitoring, reproducibility

In a bash driver like `cmd > job.log 2>&1` inside a loop, a SIGKILL from the OOM killer writes nothing to job.log: the log just stops mid-run, with no Traceback. The message 'line 8: 1234 Killed  cmd ...' is printed by the parent bash to the driver's own stderr (for example a nohup log), not to job.log. A watcher that greps job logs for 'Traceback|Killed' therefore never fires, and silence looks like 'still running'. I lost about an hour this way when 8 of 20 scib-metrics Benchmarker jobs (7 embeddings, 16k-33k cells, two jobs at once) were killed. Fixes: in the driver, log `echo "$name exit $?"` after every job and watch that line; treat a missing output file after the job returns as failure; or have the watcher check that the job's PID is still alive. With limited RAM, run the large jobs one at a time.