Skip to content

Commit 6e57e6d

Browse files
committed
DBX Storage Access docs
1 parent bad3197 commit 6e57e6d

3 files changed

Lines changed: 85 additions & 0 deletions

File tree

49.9 KB
Loading
42.8 KB
Loading

docs/snippets/cloud/integrations/databricks.mdx

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,91 @@ Then, select your authentication method:
3838
long-lived personal access tokens.
3939
</Info>
4040

41+
### Storage Access
42+
43+
Elementary requires access to the table history in order to enable automated monitors such as volume and freshness monitors.
44+
You can configure this in one of the following ways:
45+
46+
#### Option 1: Fetch history using `DESCRIBE HISTORY`
47+
48+
Elementary can fetch the table history by running `DESCRIBE HISTORY` queries on your Databricks warehouse.
49+
In the Elementary UI, choose **None** under **Storage access method**.
50+
51+
This requires `SELECT` access on the relevant tables, as described in the permissions and security section above.
52+
53+
#### Option 2: Credentials vending
54+
55+
Elementary can access the storage using temporary credentials issued by Databricks through [credential vending](https://docs.databricks.com/aws/en/external-access/credential-vending).
56+
In the Elementary UI, choose **Credentials vending** under **Storage access method**.
57+
58+
This requires granting `EXTERNAL USE SCHEMA` on the relevant schemas.
59+
60+
When using this option, Elementary only reads the Delta transaction log files from storage.
61+
62+
#### Option 3: Direct storage access
63+
64+
Elementary can access the storage directly using credentials that you configure.
65+
In the Elementary UI, choose **Direct storage access** under **Storage access method**.
66+
67+
When using this option, Elementary only reads the Delta transaction log files from storage.
68+
69+
For S3-backed Databricks storage, you can configure access in one of the following ways:
70+
71+
__AWS Role authentication__
72+
73+
<img
74+
src="/pics/cloud/integrations/databricks/storage-direct-access-role.png"
75+
alt="Databricks direct storage access using AWS role ARN"
76+
/>
77+
78+
This is the recommended approach, as it provides better security and follows AWS best practices.
79+
After choosing **Direct storage access**, select **AWS role ARN** under **Select S3 authentication method**.
80+
81+
1. Create an IAM role that Elementary can assume.
82+
2. Select "Another AWS account" as the trusted entity.
83+
3. Enter Elementary's AWS account ID: `743289191656`.
84+
4. Optionally enable an external ID.
85+
5. Attach a policy that grants read access to the Delta log files.
86+
87+
Use a policy similar to the following:
88+
89+
```json
90+
{
91+
"Version": "2012-10-17",
92+
"Statement": [
93+
{
94+
"Sid": "VisualEditor0",
95+
"Effect": "Allow",
96+
"Action": [
97+
"s3:GetObject",
98+
"s3:ListBucket"
99+
],
100+
"Resource": [
101+
"arn:aws:s3:::databricks-metastore-bucket",
102+
"arn:aws:s3:::databricks-metastore-bucket/*_delta_log*"
103+
]
104+
}
105+
]
106+
}
107+
```
108+
109+
Provide the role ARN in the Elementary UI, and the external ID as well if you configured one.
110+
111+
__AWS access keys__
112+
113+
<img
114+
src="/pics/cloud/integrations/databricks/storage-direct-access-keys.png"
115+
alt="Databricks direct storage access using AWS access keys"
116+
/>
117+
118+
If needed, you can instead provide direct AWS credentials.
119+
After choosing **Direct storage access**, select **Secret access key** under **Select S3 authentication method**.
120+
121+
1. Create an IAM user that Elementary will use for storage access.
122+
2. Enable programmatic access.
123+
3. Attach the same read-only S3 policy shown above.
124+
4. Provide the AWS access key ID and secret access key in the Elementary UI.
125+
41126
#### Access token (legacy)
42127

43128
<img

0 commit comments

Comments
 (0)