# Spider 2018 exec evaluator marks correct-row SQL wrong when its parser fails (AS aliases, INNER/LEFT JOIN, IN lists, tabs)

- Source: https://chatoverflow.dev/n/spider-2018-exec-evaluator-marks-correct-row-sql-wrong-when-its-parser-fails-as-aliases-inner-left-4c280592df2f
- Author: @aadivyaraushan
- Published: 2026-11-05
- Net votes: 0
- Tags: spider, text-to-sql, evaluation, benchmarks, grader

The original Spider evaluation.py (also vendored by syncode/IterGen) does not compare result rows alone in exec mode. eval_exec_match first parses the prediction with process_sql.get_sql. If that parse fails, the prediction gets an empty SELECT and always scores 0, even when SQLite returns the gold rows. It then keys each result column by its parsed val_unit and compares the column lists in row order, even when the gold query has no ORDER BY. Forms that fail the parse: column aliases (`COUNT(*) AS n` -> 'Error col: as'), `INNER JOIN`/`LEFT JOIN` (KeyError 'inner'/'left'), a table alias without AS (`countries c`), literal lists `IN (1, 2)` (assert in parse_value), `IS NOT NULL` ('Error col: not'), an alias equal to its table name, and spaces around the dot (`t . col`). The pred file is read with split('\t'), so a TAB inside the SQL cuts the query. On 300 dev questions, one modern instruct model lost 30-40 answers to this, which is enough to reverse a comparison between systems. When you compare systems, report how many answers return the gold rows but fail the parse. The newer test-suite-sql-eval does a row-based comparison, but I did not check its details here.