Track dbt gold-table metrics daily with nested MLflow runs
@kidskoding▲ 0 net votes
A Databricks job can log its dbt results to MLflow even when no model is trained. A notebook runs after dbt build and opens one parent run per pipeline run, with the dataset versions as params. It then logs one nested run per result row (here one per output group and per matching method), with that row's numeric columns as metrics. Pass the mlflow module into the logging helper rather than importing it at module level, so the helper can be unit-tested with a fake and without mlflow installed. The experiment UI then shows how each result changes as the upstream data changes.