Flaky container test that fails once and never reproduces: check for a silent apt-get install and "command not found" in the log
Pattern seen in a bash test suite run in fresh debian:stable-slim containers. The suite installed its dependencies with command -v ps >/dev/null || { apt-get -qq update >/dev/null 2>&1; apt-get -qq install -y procps time python3 >/dev/null 2>&1; } and never checked the result. One run failed one check, and many reruns passed.
The failing run's log had dozens of ps: command not found lines. The mirror or network had failed in that container, so procps was never installed. Code under test that finds its child processes with ps -eo pid,pgid,... got an empty list, treated the process group as missing, and aborted. Reruns passed because apt-get worked.
The same missing tool also made another check pass without testing anything. ! ps -eo args= | grep -q myproc ("no leftover process") is true when ps does not exist, because grep reads empty input. Any "tool -> grep -c" safety gate fails open in the same way (v=$(ps ... | grep -ci verdi) gives 0).
Fixes:
- After the install step, check every required tool again and exit with a distinct code (for example 2, "deps missing") instead of running the tests.
- Better, use a prebuilt image with the tools already installed, so runs do not depend on the network.
- Write negative process checks so they need a working tool, e.g. command -v ps && ! ps ... | grep -q x, or scan /proc/*/cmdline.
When a one-off failure won't reproduce, grep its log for command not found / No such file before assuming a race.