Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions docs/snippets/cloud/integrations/databricks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,38 @@ This guide contains the necessary steps to connect a Databricks environment to y
In the Elementary platform, go to Environments in the left menu, and click on the "Create Environment" button.
Choose a name for your environment, and then choose Databricks as your data warehouse type.

Provide the following fields in the form:
Provide the following common fields in the form:

- **Server Host**: The hostname of your Databricks account to connect to.
- **Http path**: The path to the Databricks cluster or SQL warehouse.
- **Access token**: The token you generated for the Elementary service principal (see step 7 under "Create service principal" above)
- **Catalog (optional)**: The name of the Databricks Catalog.
- **Elementary schema**: The name of your Elementary schema. Usually `[your dbt target schema]_elementary`.

Then, select your authentication method:

#### OAuth (M2M) — Recommended

<img
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1774307890/databricks_form_oauth_m2m_v0jhp9.png"
alt="Authenticate with M2M OAuth"
/>

- **Client ID**: The Application (client) ID of the service principal (the "Application ID" you copied in [step 5](#create-service-principal)).
- **Client secret**: The OAuth secret you generated for the service principal (see [step 7](#create-service-principal)).

<Info>
OAuth machine-to-machine (M2M) authentication is the recommended method for connecting to Databricks.
It uses short-lived tokens that are automatically refreshed, providing better security compared to
long-lived personal access tokens.
</Info>

#### Access token (legacy)

<img
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1774307890/databricks_form_token_dluwcr.png"
alt="Authenticate with Access Token"
/>

- **Access token**: A personal access token generated for the Elementary service principal.

<IpAllowlist />
39 changes: 31 additions & 8 deletions docs/snippets/dwh/databricks/create_service_principal.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,38 @@ to access your Databricks instance.
alt="Add databricks SQL access"
/>

6. Next, you may also need to allow Token Usage for this service principal (if it is not allowed for all users). To do so, under the settings menu choose Advanced -> Personal Access Tokens -> Permission Settings.
Then add the service principal there.
6. Next, generate credentials for your service principal. Choose one of the following methods:

<img
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1763316575/databricks_06_token_usage_eufjwv.png"
alt="Add databricks SQL access"
/>
**Option A: Generate an OAuth secret (Recommended)**

On the service principal page, go to the *Secrets* tab and click *Generate secret*. Copy the **Client ID** (this is the same as the "Application ID" from step 5) and the generated **Client secret** — you will need both when configuring the Elementary environment.

{/* TODO: Add screenshot of Databricks service principal Secrets tab with "Generate secret" button */}
<img
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1774307890/databricks_m2m_oauth_1_lzix16.png"
alt="Generate OAuth M2M Secret (1)"
/>
<img
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1774307890/databrick_m2m_oauth_2_yhwws8.png"
alt="Generate OAuth M2M Secret (2)"
/>

<Info>
OAuth secrets are the recommended authentication method. They enable short-lived token generation
with automatic refresh, providing better security than long-lived personal access tokens.
</Info>

**Option B: Create a personal access token (legacy)**

In order to generate a personal access token for your service principal, you may first need to allow Token Usage for it.
To do so, go to the settings menu and choose Advanced -> Personal Access Tokens -> Permission Settings, then make sure the service principal is in the list.

<img
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1763316575/databricks_06_token_usage_eufjwv.png"
alt="Allow token usage for service principal"
/>

7. Create a personal access token for your service principal. For more details, please click [here](https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-service-principals)
Then, create a personal access token for your service principal. For more details, please click [here](https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-service-principals).

8. Finally, in order to enable Elementary's automated monitors feature, please ensure [predictive optimization](https://docs.databricks.com/aws/en/optimizations/predictive-optimization#enable-or-disable-predictive-optimization-for-your-account) is enabled in your account.
7. Finally, in order to enable Elementary's automated monitors feature, please ensure [predictive optimization](https://docs.databricks.com/aws/en/optimizations/predictive-optimization#enable-or-disable-predictive-optimization-for-your-account) is enabled in your account.
This is required for table statistics to be updated (Elementary relies on this to obtain up-to-date row counts)
Loading