CTest fixtures: skip self-checking tests when basics fail
When a language's feature tests check themselves (written in the language, using its own assert and equality, run by its own test runner), a bug in those basics can make every feature test pass silently, or fail all at once. Keep a small tier checked from outside, by exit codes and the runner's exact output, and gate the rest with CTest fixtures. Give the basic tests FIXTURES_SETUP basics; with Catch2 that's catch_discover_tests(target PROPERTIES FIXTURES_SETUP basics), which tags every discovered test. Give the feature tests FIXTURES_REQUIRED basics. If any setup test fails, CTest marks the dependents '***Not Run' instead of running them. Running one dependent with -R also pulls in all its setup tests.
Confirmed with CTest 3.28.3 and plain add_test (I didn't test the Catch2 catch_discover_tests part). Two setup tests with FIXTURES_SETUP basics, two feature tests with FIXTURES_REQUIRED basics, one unrelated test. With one setup test failing:
basic_bad ***Failed, thenFailed test dependencies: basic_badandfeat_a ***Not Run,feat_b ***Not Run, while the unrelated test still ran. Note that Not Run counts as a failure: "40% tests passed, 3 tests failed out of 5", ctest rc=8. So CI goes red with the root cause listed first, not silently green.ctest -R feat_aran basic_ok, basic_bad and feat_a (3 tests).ctest -R feat_a -FS basicsskips the setup tests and runs only feat_a, which is handy while iterating on one feature test. Gotcha from writing the repro:$ENV{X}inside add_test's COMMAND is expanded at configure time, not when the test runs.