Have you taken this into account - The Databricks docs state "Starting with Databricks Runtime 13.0 %pip commands do not automatically restart the Python process. If you install a new package or update an existing package, you may need to use dbutils.library.restartPython() to see the new packages."

• https://docs.databricks.com/aws/en/libraries/notebooks-python-libraries#manage-libraries-with-pip-commands
• https://docs.databricks.com/aws/en/libraries/restart-python-process

Also have you considered the recommendation on driver node sizes for notebook scoped libraries

I didn't take into account the recommendation to restart python process, thanks for that. I will refactor this draft PR to make it more production ready (avoiding duplicated code) and consider that.

About the how large should the driver node be, this solution is agnostic about cluster size. It will depend on the team that previously defined the cluster (no matter if it was an all purpose cluster, job cluster, whatever).

@fedemgp can you please address the review comments.

@fedemgp can you please add the functional tests?

Hello @tejassp-db, apologies for the delayed response — I was out of the office. I've now pushed the requested changes.

Hello @tejassp-db, were you able to take a look at the latest changes?

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks/python_models
python_config.py
python_submissions.py					31
Project Total

This report was generated by python-coverage-comment-action

Hi @fedemgp sorry for the late reply, I was on extended leave. Thanks for the functional test + fixes. I see the tests are failing though. The fix for this should be quite simple

• Both model fixtures in fixtures.py are missing return df — without it dbt can't materialize the result and the tests fail. Add return df at the end of both notebook_scoped_packages_cmd_api_model and notebook_scoped_packages_notebook_run_model
• TestNotebookScopedPackagesCommandAPI references the wrong fixture — it uses notebook_scoped_packages_notebook_run_model (which has create_notebook=True) instead of notebook_scoped_packages_cmd_api_model (which has create_notebook=False). This means both test classes exercise the same notebook-upload path and the Command API path has no coverage.

You can refer to the fixes here (verified running against a live cluster).

I see the tests are failing though

The failing github action run should have more details on the failure reason

I see the tests are failing though

The failing github action run should have more details on the failure reason

You are totally right. I thought I had run the tests locally, but I guess I was wrong. I got confused because the PR skipped the step. I've fixed them now and ran them locally. My tests passed, but some unrelated ones failed. I think they might be failing due to a lack of configuration in my dev environment.

I pushed a new commit applying the fixes you recommended. Thanks!

Changes look good, thanks for the PR.

Glad to hear it, do you have an ETA of when the change will be released?