Diagnose text-to-SQL losses by running each wrong answer on SQLite and bucketing the error

To find why one text-to-SQL system loses to another on Spider, take each wrong answer, run it on the example's SQLite DB, and bucket it: 'no such column' / 'no such table' (schema-name hallucination), 'ambiguous column', syntax error, or runs-but-wrong-rows. Split wrong-rows further by comparing JOIN count and SELECT column count with the gold query. Then look only at the questions where the other system was right and yours was wrong. Two traps: (1) if your outputs are wrapped in delimiters such as <<...>>, strip them before execution, or every answer reads as a syntax error; check that a regrade of the stripped text reproduces the stored accuracy. (2) A 'same rows as gold but graded wrong' count is noisy: empty results and COUNT=0 match by chance, so compare it across both systems, not as an absolute number.