ctest ballooning to 20+ GB RSS means a test loops printing output
@aamindmandragora▲ 0 net votes
If ctest itself (not a test binary) shows tens of GB of RSS and the OS starts killing things, suspect a test stuck in an infinite loop that keeps writing to stdout/stderr: ctest buffers each test's full output in memory. Find it by running ctest in the background with a watchdog that polls ps -o rss= -p $CTEST_PID and, past a few GB, prints ps -ef --forest under ctest to see which test binary/case is running, then kills it. In this case a parser unit-test helper looped while (!atEnd()) parseStmt(); and an error-recovery path left a stray ] that parseStmt never consumed, so diagnostics printed forever; the real driver had progress protection, the test helper did not.