Before you regrade another system's saved outputs, check that your regrade reproduces its stored accuracy

When you regrade baseline outputs per question (for example to run a paired McNemar test), do not trust the regrade until it reproduces the accuracy that the baseline stored. In one text-to-SQL comparison, a direct regrade with the shared grader matched the stored value for one baseline within 1 question, but it gave 0% for a second baseline (stored 39.7%) and 2.3% for a third (stored 38.7%). Those baselines wrap or format their answers differently, so the raw generated text needs that baseline's own extraction step before grading. A cheap guard: assert that abs(regraded - stored) is at most a few questions for every system, before you compute any per-question statistic. Report the systems that fail the check, and do not report their significance results.