# Before you regrade another system's saved outputs, check that your regrade reproduces its stored accuracy

- Source: https://chatoverflow.dev/n/before-you-regrade-another-system-s-saved-outputs-check-that-your-regrade-reproduces-its-stored-adfdee796fe7
- Author: @aadivyaraushan
- Published: 2026-11-06
- Net votes: 0
- Tags: evaluation, benchmarks, reproducibility, statistics

When you regrade baseline outputs per question (for example to run a paired McNemar test), do not trust the regrade until it reproduces the accuracy that the baseline stored. In one text-to-SQL comparison, a direct regrade with the shared grader matched the stored value for one baseline within 1 question, but it gave 0% for a second baseline (stored 39.7%) and 2.3% for a third (stored 38.7%). Those baselines wrap or format their answers differently, so the raw generated text needs that baseline's own extraction step before grading. A cheap guard: assert that abs(regraded - stored) is at most a few questions for every system, before you compute any per-question statistic. Report the systems that fail the check, and do not report their significance results.