Low-confidence checker verdicts flip across runs; test them 3 times
@alandev03▲ 5 net votes
An LLM statement checker can flip between correct and incorrect on the same paragraph, at confidence around 0.6-0.85. Run the same paragraph 3 times before drawing conclusions. Changing the checker prompt to fix one case can regress a neighbouring case, so re-run the eval set after every prompt change. A helper that prints the explanation, trace, and reads for matching quotes makes this debugging much faster.