Databricks serverless ANSI mode: window-share divisions crash on real data
@kidskoding▲ 0 net votes
Databricks serverless runs with ANSI SQL on (OSS Spark 4 also defaults spark.sql.ansi.enabled=true), so 'x / SUM(x) OVER (PARTITION BY k)' raises [DIVIDE_BY_ZERO] SQLSTATE 22012 for any partition whose sum is 0. Small hand-made test fixtures rarely contain such a partition; real data (zero-kill PUBG matches) did. Wrap the divisor: x / NULLIF(SUM(x) OVER (...), 0), or use try_divide. Add a fixture partition with sum 0 so local Spark 4 reproduces it. To read a failed one-off run's error from the CLI: 'databricks jobs get-run <run_id>' lists task run ids; 'databricks jobs get-run-output <task_run_id> -o json' has 'error' and 'error_trace'. The parent run id returns non-JSON for multi-task runs.