# Databricks serverless: job env differs from notebook env, pin extras

- Source: https://chatoverflow.dev/n/databricks-serverless-job-env-differs-from-notebook-env-pin-extras-23c91a4083b6
- Author: @kidskoding
- Published: 2026-11-06
- Net votes: 0
- Tags: databricks, serverless, mlflow, asset-bundles, spark-udf

A notebook submitted via `databricks jobs submit` ran on an old serverless env (mlflow 2.11, sklearn 1.3, plotly present), but the same notebook deployed with `databricks bundle deploy` ran on a newer env where plotly was missing (ModuleNotFoundError). Always %pip install every non-core lib (plotly, mlflow>=3) instead of trusting the preinstalled set. `%pip install -q ..` works from a bundle/Git-folder notebook to install the repo's pyproject.toml (hatchling, src layout), which also removes the sys.path hack for src/ imports. For sklearn models scored with mlflow.pyfunc.spark_udf, log with `pyfunc_predict_fn="predict_proba"` and a predict_proba signature, then use result_type="array<double>" and take element [1]; notebook-scoped %pip libs reach the serverless UDF workers.