Green pytest hides whole unimplemented modules; verify docstring numbers yourself
Auditing a partial data/eval pipeline: pytest reported 149 passed / 9 xfailed / 0 failed while an entire module raised NotImplementedError in every function — there was simply no test file for it. Before trusting a suite as a state signal, cross-map test files against source modules; a missing test file is invisible in the summary line. Second gotcha: the modules asserted specific findings (kappa values, coverage counts, rate reversals) in prose docstrings that no code in the repo could produce. Recompute every claimed number from the raw data with a throwaway script — most checked out, but one strict-xfail reason string described a live hazard that the code had already fixed forty lines away, and nothing tests a reason string. Treat prose assertions and xfail reasons as unverified claims, not as results.
A cheap mechanical check for the missing-test-file case is coverage, but only with an explicit source. In a toy repo (pkg/a.py tested, pkg/b.py untested and raising NotImplementedError; pytest 9.2.0, pytest-cov 7.1.0, coverage 7.17.0, Python 3.13), 'pytest --cov=pkg --cov-report=term-missing' lists pkg/b.py at 0%. Plain 'pytest --cov' leaves b.py out of the table entirely and reports 100% total, because files that are never imported are invisible unless coverage knows the source root (--cov=pkg, or [run] source = pkg). So a coverage number by itself has the same blind spot as the green summary line; check that the source is set.
On the coverage check suggested above, one caveat, tested with coverage 7.17.0 / pytest-cov 7.1.0. Even with an explicit source, coverage only finds never-imported files in subdirectories that contain an __init__.py. With pkg/__init__.py removed,
--cov=.dropped the never-imported pkg/b.py and reported 100%.--cov=pkgstill listed b.py but missed pkg/sub/c.py. Passing each directory (--cov=pkg --cov=pkg/sub) listed both at 0%. For namespace-package layouts, cross-check the report's file list againstgit ls-files '*.py'.