Test runner scripts: a printed FAIL, a sanitizer report, or a trailing echo can all leave exit status 0
An independent review of a C++ glue test script (bash, many arms, plain and ASan/UBSan/TSan builds) found three ways it could report success while tests failed. The author's '0 fails' record was therefore not evidence.
1. A comparison against a frozen reference printed 'FAIL ...' but didn't increment the fail counter, because one branch only echoed.
2. Sanitizer findings were never turned into failures. ASan and UBSan print a report but, depending on options, the process can still exit 0. UBSan in particular continues by default, and TSan warnings don't change the exit code unless halt_on_error or exitcode is set.
3. The script ended with echo "done: $fails failures", so its exit status was echo's 0, not [ $fails -eq 0 ].
Fixes:
- Count every arm you expect to run and fail if the count of arms that ran differs. That catches arms that never ran.
- Exit non-zero on any FAIL line, and make the last line exit $(( fails > 0 )).
- Run sanitizers with ASAN_OPTIONS=halt_on_error=1:exitcode=23, UBSAN_OPTIONS=halt_on_error=1:print_stacktrace=1 (and build with -fno-sanitize-recover=all), and TSAN_OPTIONS=halt_on_error=1:exitcode=66. Also grep the logs for 'ERROR: AddressSanitizer', 'runtime error:' and 'WARNING: ThreadSanitizer' as a backstop.
- Add injected-fault self-tests: a wrong reference byte, a deliberate leak or UB, a skipped arm. Each must make the runner exit non-zero. Reviewers should rerun the suite and check those, not trust a printed summary.