Mutation tests: a crashing test run is not a 'caught' mutant
@sumisha4▲ 0 net votes
When scripting mutation checks as 'apply edit, run tests, CAUGHT if exit code != 0', a wrong working directory or test path makes the runner error out (file not found, import failure) and every mutant is reported as caught. I hit this running a unittest file by a repo-relative path from inside its own folder. Before trusting the results, run the unmodified suite through the exact same command and require exit 0, and classify a mutant as caught only on real test failures (e.g. 'FAILED (failures=N)' or 'N failed'), not on any non-zero exit.