Databricks CLI as possible connection type in profile config

[Rhiyo]

Describe the feature

Adding databricks cli connection support to profile connection config.

Describe alternatives you've considered

using oauth as the auth_type.

Additional context

When using Databricks Connect v2 or DBX SQL Connector I can use a Databricks CLI to connect to databricks. This allows Databricks CLI to manage and connection decoupling it nicely and making so I do not have to reauthenticate between usage. Currently if using dbt the browser will reopen to create it's own authentication with oauth. If it supported the CLI it wouldn't need to reauthenticate.

You could just have the auth_type option as CLI - but you'd need two other new config paremters:

cli_path the path to the cli exe

cli_profile the dbx cli profile to use

Dbt-databricks is already using the DBX connection Config object and these above values can go directly into it.

Who will this benefit?

Anyone using multiple different libraries in their processes to connect to databricks via oauth locally.

Are you interested in contributing this feature?

Yes, I am very across the underlying DBX connection logic and I am able to support on this.

[ferdyh]

This would be very usefull

[Rhiyo]

Databricks has the unified authentication config object these days. It'd be great if we could stay in line with that then we wouldn't have to have to update this connection's code each time.

I feel like this connection should just take a connection_config like param and pass all the variables into the DBX unified config object. I did something similar for soda: sodadata/soda-core#2220