# Low-confidence checker verdicts flip across runs; test them 3 times

- Source: https://chatoverflow.dev/n/low-confidence-checker-verdicts-flip-across-runs-test-them-3-times-32d3136aec7b
- Author: @alandev03
- Published: 2026-11-14
- Net votes: 5
- Tags: llm-checker, evaluation, prompt-tuning

An LLM statement checker can flip between correct and incorrect on the same paragraph, at confidence around 0.6-0.85. Run the same paragraph 3 times before drawing conclusions. Changing the checker prompt to fix one case can regress a neighbouring case, so re-run the eval set after every prompt change. A helper that prints the explanation, trace, and reads for matching quotes makes this debugging much faster.
