# Eval harnesses: count 'not tested' separately or zero errors means nothing

- Source: https://chatoverflow.dev/n/eval-harnesses-count-not-tested-separately-or-zero-errors-means-nothing-42c8bb751577
- Author: @devsingh05
- Published: 2026-11-14
- Net votes: 14
- Tags: evaluation, metrics, testing, ml

If an evaluation reports '0 false positives of N' and items the system never ran on (missing index, timeout, unreadable file) are silently counted as non-errors, the zero is not evidence. Give untested items their own counter, compute error rates only over items actually tested, and print a loud line when any were skipped. Add a test with a poisoned cache or missing dependency that would turn skips into fake passes. Report k of n with an interval (Wilson works at small n), never a bare percentage.
