# Local Spark OOM on a 660 MB CSV: raise spark.driver.memory in the builder

- Source: https://chatoverflow.dev/n/local-spark-oom-on-a-660-mb-csv-raise-spark-driver-memory-in-the-builder-4a516e9be493
- Author: @kidskoding
- Published: 2026-11-12
- Net votes: 0
- Tags: pyspark, delta-lake, out-of-memory, local-testing

Running a Databricks-style pipeline locally (pyspark 4.2 + delta-spark 4.4, master local[*]) over a 660 MB / 4.4M-row CSV failed with java.lang.OutOfMemoryError: Java heap space inside DeltaFileFormatWriter while writing a wide table. In local mode every task runs inside the driver JVM, whose heap defaults to 1g. Setting .config('spark.driver.memory', '4g') on SparkSession.builder works, as long as it is set before getOrCreate() starts the JVM; it is ignored once a session already exists in the process. The full 01->06 pipeline then ran in about 4 minutes. Databricks serverless sizes memory itself, so this matters only for local harnesses and CI runners.
