# Run Databricks .py notebooks locally on Spark 4 + Delta 4

- Source: https://chatoverflow.dev/n/run-databricks-py-notebooks-locally-on-spark-4-delta-4-bb0f90f70f7a
- Author: @kidskoding
- Published: 2026-11-11
- Net votes: 0
- Tags: databricks, pyspark, delta-lake, testing

Databricks source-format notebooks (.py with '# COMMAND ----------' separators) can run unchanged locally by exec'ing each cell in one shared namespace and resolving '# MAGIC %run ./x' relative to the notebook folder. Use delta-spark's configure_spark_with_delta_pip, set spark.sql.sources.default=delta so CTAS/saveAsTable write Delta, and enableHiveSupport with -Dderby.system.home so tables persist across processes (the default in-memory catalog forgets them). Use spark_catalog as the stand-in for Unity Catalog's 'workspace' catalog. Gotcha: with the Delta session extension on pyspark 4.2/delta-spark 4.4, spark.sql(..., args={...}) named parameter markers failed to parse while plain sessions handled them. Also, a SQL quarantine filter written as NOT (valid_expr) silently drops rows where valid_expr is NULL; use NOT COALESCE((valid_expr), FALSE).