podman logs <id> after a --rm container exits: 'Error: no container with ID ... found in database: no such container'; the output is gone, tee it or drop --rm
Fix: if you need the output of a long batch container after it exits, either don't use --rm (remove it yourself after reading the logs) or capture stdout at run time:
podman run --name sfm -v ./output:/app:Z drone-sfm python3 /opt/pipeline.py # no --rm
podman logs sfm > run.log; podman rm sfm
# or
podman run --rm -v ./output:/app:Z drone-sfm python3 -u /opt/pipeline.py 2>&1 | tee output/run.logSymptom: a COLMAP structure-from-motion pipeline on 100 drone photos ran as podman run -d --rm .... A follow-up sleep 180 && podman logs <id> returned
Error: no container with ID 476f3484eabf... found in database: no such container.
The output folder held downloaded images, gps_data.txt and an empty sparse/, with no clue why COLMAP stopped. The whole download and feature-extraction run had to be repeated in the foreground with | tee run.log to see that feature extraction was being OOM-killed (fixed by smaller images, --SiftExtraction.max_num_features 4096 and --SiftExtraction.num_threads 4).
Cause: --rm removes the container, and with it the stored log, as soon as the main process exits, whether it succeeded or crashed. podman ps -a won't list it either. Exit status, OOM-kill information (podman inspect --format '{{.State.OOMKilled}}') and logs are all lost.
Useful variants: podman run -d --name job ... then podman wait job; podman logs job; podman inspect job --format '{{.State.ExitCode}} {{.State.OOMKilled}}'; podman rm job. Or add --log-driver k8s-file --log-opt path=./job.log (podman) so the log file is written to a path you choose. Combine this with PYTHONUNBUFFERED=1 so the log isn't empty when the process is killed.