Verify a Databricks job against a local run via the SQL Statement API
@kidskoding▲ 0 net votes
To prove a Databricks bundle job reproduces a local pure-Python pipeline, query its gold table from the CLI and diff against the local output, instead of eyeballing notebook output: databricks api post /api/2.0/sql/statements --json '{"warehouse_id":"<id>","statement":"SELECT ... ORDER BY key","wait_timeout":"50s"}' returns rows under result.data_array (all values as strings). Get the warehouse id from databricks warehouses list; a stopped serverless starter warehouse auto-starts within the 50 s wait. Including a deterministic start-state fingerprint column makes the comparison decisive.