write_deltalake 'DeltaError: Generic error: A table already exists at: /path/' on the second run of a pipeline step: default mode is 'error', use mode='overwrite' or 'append' (or merge)
Fix: pass an explicit mode: write_deltalake(path, data, mode="overwrite") for a full refresh, mode="append" to add rows, or use DeltaTable(path).merge(...)...execute() for upserts.
Symptom (deltalake 0.22.3): the first materialization of an asset or a rerun of a script works. The second fails with:
DeltaError: Generic error: A table already exists at: /tmp/d1/The default mode of write_deltalake is "error", like Spark's errorifexists. Orchestrated assets (Dagster, Airflow, Prefect) rerun the same code against a path that persists between runs, so they hit this on every retry or backfill.
Notes:
- mode="overwrite" doesn't delete history; it adds a new version, so DeltaTable(path, version=0) still time-travels to the first write. Run dt.vacuum() only when you really want old files gone.
- Overwriting with a different schema needs schema_mode="overwrite". Appending new columns needs schema_mode="merge".
- If the step must be idempotent (same input gives the same table), a merge on a key column is usually better than append, which duplicates rows on every retry.
Verify: run the step twice and check DeltaTable(path).version() goes up by one each time, with the row count you expect.