Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/cloud/integrations/bi/hex.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,18 @@ badge: "Context Engine"
After you connect Hex, Elementary will automatically and continuously extend the lineage to the project & cell level.
This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures.

### Prerequisites

The Hex integration relies on **query history** to discover which tables Hex projects query. It is supported on **BigQuery** and **Snowflake**.

Elementary's service account must have permissions to read query history. Refer to your warehouse's setup page for the required roles:
- [BigQuery permissions](/cloud/integrations/dwh/bigquery#grant-service-user-access)
- [Snowflake permissions](/cloud/integrations/dwh/snowflake)

<Note>
If Elementary cannot read query history, the Hex integration will not discover any exposures.
</Note>

### Create API Token

Elementary needs a workspace token in your account in order to access Hex's API on your behalf. <br/>
Expand All @@ -20,6 +32,15 @@ Choose the Hex connection and provide the following details to validate and comp
- **Base URL**: Your Hex workspace URL. For example: `https://app.hex.tech/my-workspace/`
- **Workspace Token**: The Hex workspace token you've created on the previous step.

### Hex queries on a different warehouse environment

By default, Elementary reads query history from the warehouse environments (e.g., BigQuery projects, Snowflake accounts) referenced in your dbt project. If Hex executes queries through a **different environment** (e.g., a dedicated Hex execution environment), you will need to:

1. **Grant query history permissions** — the same permissions required for your primary environment. See the [prerequisites](#prerequisites) above for links to your warehouse's permissions page.

2. **Let us know** — contact Elementary support with the environment details so we can configure Elementary to also read query history from it.

Without both steps, Elementary will not find any Hex queries in that environment and no Hex exposures will appear.

### Limitations

Expand Down
4 changes: 4 additions & 0 deletions docs/snippets/cloud/integrations/bigquery.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ Select a tab below and follow the steps for your chosen method.

<PermissionsAndSecurity />

For BigQuery, this translates to:
- **BigQuery Data Viewer** on the Elementary dataset (for reading Elementary schema data).
- **BigQuery Metadata Viewer** and **BigQuery Resource Viewer** at the **project level** (for reading `INFORMATION_SCHEMA.JOBS` query history and table metadata). These project-level roles are required for BI integrations like Hex that rely on query history.

### Fill the connection form

Use the **Authentication method** toggle at the top of the form to select either **Service account** or **Workload Identity Federation**, matching the method you set up above. The credentials upload field changes based on your selection:
Expand Down
6 changes: 3 additions & 3 deletions docs/snippets/cloud/integrations/permissions-and-security.mdx
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#### Permissions and security

Elementary cloud doesn't require read permissions to your tables and schemas, but only the following:
Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following:

- Read-only access to the elementary schema.
- Access to read metadata in information schema and query history, related to the tables in your dbt project.
- Read-only access to the Elementary schema.
- Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project.

It is recommended to create a user using the instructions specified above to avoid granting excess privileges.
For more details, refer to [security and privacy](/cloud/general/security-and-privacy).
44 changes: 33 additions & 11 deletions docs/snippets/dwh/bigquery/grant_user_access_on_dataset_level.mdx

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Pre-existing images in grant_user_access_on_dataset_level.mdx use external hosting and lack Frame wrappers

The three images in docs/snippets/dwh/bigquery/grant_user_access_on_dataset_level.mdx:40-43, 47-49, 57-59 use Cloudinary URLs (external hosting) and are not wrapped in <Frame> components. Both of these violate rules in docs/skills.md:198-209 ("All images must be wrapped in a <Frame> with a caption. No exceptions." and "Store images as local files under docs/pics/."). These are pre-existing issues not introduced by this PR — the images are in unchanged context lines — so they are not reported as PR bugs, but they should be cleaned up in a follow-up.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
@@ -1,15 +1,39 @@
### Grant service user access to specific datasets
### Grant service user access

In order for a service user to work with Elementary cloud, it requires the following permissions:
In order for a service user to work with Elementary Cloud, it requires the following permissions:

- Role "BigQuery Data Viewer" on your Elementary dataset.
- Roles "BigQuery Metadata Viewer", "BigQuery Resource Viewer" on the entire project and any external sources it is referencing.
- Role **"BigQuery Data Viewer"** on your Elementary dataset.
- Roles **"BigQuery Metadata Viewer"** and **"BigQuery Resource Viewer"** at the **project level** for the project containing your dbt models and any external projects it references.

To grant a role on a specific dataset, follow these steps:
<Note>
"BigQuery Metadata Viewer" and "BigQuery Resource Viewer" **must be granted at the project level** (via IAM & Admin), not at the dataset level. These roles include permissions such as `bigquery.jobs.listAll` which are required for reading query history and metadata from `INFORMATION_SCHEMA` — they cannot be granted on individual datasets.
</Note>

1. Go to your project in [BigQuery console](https://console.cloud.google.com/bigquery)
#### Step 1: Grant project-level roles

2. In the "Explorer" tab, find your desired dataset.
These roles are required for Elementary to read query history (`INFORMATION_SCHEMA.JOBS`) and table metadata. Without them, BI integrations (such as Hex) that rely on query history will not function.

1. Go to [IAM & Admin](https://console.cloud.google.com/iam-admin/iam) in the Google Cloud Console.

2. Select the project that contains your dbt models.

3. Click **"Grant Access"** at the top.

4. Fill out the form:
- In the "New principals" field, enter the **email address** of the Elementary service account.
- Add the role **"BigQuery Metadata Viewer"** (`roles/bigquery.metadataViewer`).
- Click **"Add another role"** and add **"BigQuery Resource Viewer"** (`roles/bigquery.resourceViewer`).
- Click **"Save"**.

5. **Repeat for any additional GCP projects** referenced in your dbt sources or used by BI tools (e.g., Hex, Looker) for query execution.

#### Step 2: Grant dataset-level role on the Elementary dataset

This role provides read access to your Elementary schema data.

1. Go to your project in [BigQuery console](https://console.cloud.google.com/bigquery).

2. In the "Explorer" tab, find your Elementary dataset.

3. Click on the three dots icon next to the dataset name, then Share.

Expand All @@ -26,13 +50,11 @@ To grant a role on a specific dataset, follow these steps:
/>

5. Fill out the form:
- In the "New principals" textbox, write the **email address** of your user.
- In the "Select a role" dropdown menu, choose the desired role (BigQuery Data Viewer for your Elementary dataset, BigQuery Metadata Viewer, BigQuery Resource Viewer for your dbt dataset).
- In the "New principals" textbox, write the **email address** of your service account.
- In the "Select a role" dropdown menu, choose **"BigQuery Data Viewer"**.
- Click "Save".

<img
src="https://res.cloudinary.com/diuctyblm/image/upload/f_auto,q_auto/v1/dwh/bigquery/grant_access_form"
alt="Grant access"
/>

Make sure to grant the correct access to your Elementary dataset **and** your dbt dataset.
Loading