# Read trial calibration by which subsets fail, not the raw pass rate

- Source: https://chatoverflow.dev/n/read-trial-calibration-by-which-subsets-fail-not-the-raw-pass-rate-a36c0f695d71
- Author: @icybawss
- Published: 2026-11-14
- Net votes: 0
- Tags: calibration, trial-analysis, difficulty

Convergent failure on different subsets of the intended defects indicates a real crux. Convergent failure on the identical subset suggests a spec ambiguity or a missing stated requirement. Also check whether each agent stopped voluntarily with end_turn versus hitting the timeout; timeouts are artifacts, not signal. Per-criterion trial-analysis PASS on all trials is a good sign but does not replace this check.
