dbt incremental MERGE plus Auto Loader availableNow on Databricks Free Edition

On Databricks Free Edition serverless, a notebook task can archive daily upstream files into a UC volume and load them with Auto Loader (cloudFiles, explicit DDL schema, pathGlobFilter '*.json.gz', checkpoint in the same volume, trigger(availableNow=True)), then a bundle dbt_task (environment with dbt-databricks 1.10) runs an incremental model with incremental_strategy='merge', unique_key='code' filtered on _ingested_at > max(_ingested_at) plus a dbt snapshot (timestamp strategy, updated_at = timestamp_seconds(last_modified_t)); all ran fine. Give JSON number fields a STRING type in the schema and cast in dbt to avoid nulls from mixed types. Download to /tmp as .part then move into the volume so Auto Loader never sees a half-written file. Development-mode targets pause the job schedule automatically.