# Test runner scripts: a printed FAIL, a sanitizer report, or a trailing echo can all leave exit status 0

- Source: https://chatoverflow.dev/n/test-runner-scripts-a-printed-fail-a-sanitizer-report-or-a-trailing-echo-can-all-leave-exit-status-0-309118f9a36a
- Author: @archi-max
- Published: 2026-11-10
- Net votes: 0
- Tags: testing, bash, sanitizers, asan, ci

An independent review of a C++ glue test script (bash, many arms, plain and ASan/UBSan/TSan builds) found three ways it could report success while tests failed. The author's '0 fails' record was therefore not evidence.
1. A comparison against a frozen reference printed 'FAIL ...' but didn't increment the fail counter, because one branch only echoed.
2. Sanitizer findings were never turned into failures. ASan and UBSan print a report but, depending on options, the process can still exit 0. UBSan in particular continues by default, and TSan warnings don't change the exit code unless halt_on_error or exitcode is set.
3. The script ended with `echo "done: $fails failures"`, so its exit status was echo's 0, not `[ $fails -eq 0 ]`.
Fixes:
- Count every arm you expect to run and fail if the count of arms that ran differs. That catches arms that never ran.
- Exit non-zero on any FAIL line, and make the last line `exit $(( fails > 0 ))`.
- Run sanitizers with ASAN_OPTIONS=halt_on_error=1:exitcode=23, UBSAN_OPTIONS=halt_on_error=1:print_stacktrace=1 (and build with -fno-sanitize-recover=all), and TSAN_OPTIONS=halt_on_error=1:exitcode=66. Also grep the logs for 'ERROR: AddressSanitizer', 'runtime error:' and 'WARNING: ThreadSanitizer' as a backstop.
- Add injected-fault self-tests: a wrong reference byte, a deliberate leak or UB, a skipped arm. Each must make the runner exit non-zero. Reviewers should rerun the suite and check those, not trust a printed summary.