Skip to content

Commit ab33562

Browse files
GuyEshdatGuyEshdat
authored
WIF docs (#2199)
1 parent 55fe8f8 commit ab33562

7 files changed

Lines changed: 117 additions & 17 deletions

File tree

‎docs/cloud/integrations/log-streaming/gcs.mdx‎

Lines changed: 35 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,40 @@ Before configuring log streaming to GCS, you'll need:
2222
- The bucket must exist and be accessible
2323
- You'll need the bucket path (e.g., `gs://my-logs-bucket`)
2424

25-
2. **Google Cloud Service Account** - A service account with permissions to write to the bucket
26-
- Required role: `Storage Object User` (roles/storage.objectUser)
27-
- You'll need to generate a service account JSON key file
28-
- The service account key file must be uploaded in Elementary
29-
- **Workload Identity Federation**: Support for Workload Identity Federation with BigQuery service accounts is coming soon
25+
2. **Authentication** - Either a Google Cloud service account or a Workload Identity Federation setup, with the **Storage Object User** (`roles/storage.objectUser`) role granted on the bucket. See [Authentication methods](#authentication-methods) below for both options.
3026

27+
## Authentication methods
28+
29+
Elementary supports two authentication methods for GCS. Pick the one that fits your security model:
30+
31+
- **Service account** — create a service account, download its JSON key, and upload the key to Elementary. Simplest to set up.
32+
- **Workload Identity Federation (WIF)** — Elementary authenticates from its AWS role through a federated identity. No long-lived credentials are stored in Elementary.
33+
34+
Select a tab below and follow the steps for your chosen method.
35+
36+
<Tabs>
37+
<Tab title="Service account">
38+
1. Go to [Google Cloud Console > IAM & Admin > Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) and create a service account (or select an existing one).
39+
40+
2. Grant the service account the **Storage Object User** (`roles/storage.objectUser`) role on your GCS bucket.
41+
42+
3. Generate a JSON key for the service account:
43+
1. Select your service account.
44+
2. Click the three dots menu and select 'Manage keys'.
45+
3. Click 'ADD KEY' and select 'Create new key'.
46+
4. Choose 'JSON' format and click 'CREATE'. The JSON file downloads automatically.
47+
48+
4. You will upload this JSON key to Elementary in the connection form below, in the **Service account file** field.
49+
</Tab>
50+
<Tab title="Workload Identity Federation">
51+
The WIF setup is identical to the one documented for BigQuery — the only GCS-specific change is the permission you grant to the pool principal. Follow the Workload Identity Federation steps on the [BigQuery connection page](/cloud/integrations/dwh/bigquery) (select the **Workload Identity Federation** tab), with the following change:
52+
53+
- In the step where you grant IAM roles to the principalSet, **do not** grant the BigQuery roles. Instead, grant the **Storage Object User** (`roles/storage.objectUser`) role on your GCS bucket. You can do this from the bucket's 'Permissions' tab: 'GRANT ACCESS' → paste the same principalSet → assign `Storage Object User`.
54+
- You can skip the dataset-level grant step entirely — it is BigQuery-specific.
55+
56+
Once you have the downloaded WIF credential configuration file, upload it to Elementary in the connection form below, in the **WIF credential file** field.
57+
</Tab>
58+
</Tabs>
3159

3260
## Configuring Log Streaming to GCS
3361

@@ -42,14 +70,8 @@ Before configuring log streaming to GCS, you'll need:
4270

4371
4. Enter your GCS configuration:
4472
- **Bucket Path**: The full GCS bucket path (e.g., `gs://my-logs-bucket`)
45-
- **Service Account Key File**: Upload your Google Cloud service account JSON key file
46-
- To generate a service account key file:
47-
1. Go to [Google Cloud Console > IAM & Admin > Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts)
48-
2. Select your service account (or create a new one)
49-
3. Click the three dots menu and select "Manage keys"
50-
4. Click "ADD KEY" and select "Create new key"
51-
5. Choose "JSON" format and click "CREATE"
52-
6. The JSON file will be downloaded automatically
73+
- **Authentication method**: Use the toggle to select **Service account** or **Workload Identity Federation**, matching the method you set up in [Authentication methods](#authentication-methods) above.
74+
- **Service account file** (Service account method) or **WIF credential file** (Workload Identity Federation method): Upload the JSON file you prepared.
5375

5476
5. Click **Save** to enable log streaming
5577

‎docs/pics/bigquery-wif/add_aws_provider.png‎

125 KB
Loading

‎docs/pics/bigquery-wif/attribute_condition.png‎

188 KB
Loading

‎docs/pics/bigquery-wif/configure_application_download_config.png‎

153 KB
Loading

‎docs/pics/bigquery-wif/grant_access_federated_identities.png‎

137 KB
Loading

‎docs/snippets/cloud/integrations/bigquery.mdx‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,41 @@
11
import CloudServiceAccount from '/snippets/dwh/bigquery/cloud_service_account.mdx';
2+
import CloudWIF from '/snippets/dwh/bigquery/cloud_wif.mdx';
23
import PermissionsAndSecurity from '/snippets/cloud/integrations/permissions-and-security.mdx';
34
import IpAllowlist from '/snippets/cloud/integrations/ip-allowlist.mdx';
45

56
This guide contains the necessary steps to connect a BigQuery environment to your Elementary account.
67

7-
<CloudServiceAccount />
8+
## Choose an authentication method
9+
10+
Elementary supports two authentication methods for BigQuery. Pick the one that fits your security model:
11+
12+
- **Service account** — create a service account, download its JSON key, and upload the key to Elementary. Simplest to set up.
13+
- **Workload Identity Federation (WIF)** — Elementary authenticates from its AWS role through a federated identity. No long-lived credentials are stored in Elementary.
14+
15+
Select a tab below and follow the steps for your chosen method.
16+
17+
<Tabs>
18+
<Tab title="Service account">
19+
<CloudServiceAccount />
20+
</Tab>
21+
<Tab title="Workload Identity Federation">
22+
<CloudWIF />
23+
</Tab>
24+
</Tabs>
825

926
<PermissionsAndSecurity />
1027

1128
### Fill the connection form
1229

13-
Provide the following fields:
30+
Use the **Authentication method** toggle at the top of the form to select either **Service account** or **Workload Identity Federation**, matching the method you set up above. The credentials upload field changes based on your selection:
31+
32+
- **Service account file** (Service account method): the service account JSON key file you downloaded.
33+
- **WIF credential file** (Workload Identity Federation method): the external account JSON configuration file you downloaded from the identity pool.
34+
35+
Then provide the remaining fields:
1436

15-
- **Service account file**: The service account file you generated for Elementary. For more information, refer to [Create BigQuery service account](./bigquery#create-bigquery-service-account).
1637
- **Project**: The name of your BigQuery project.
1738
- **Elementary dataset**: The name of your Elementary dataset. Usually `[dataset name]_elementary`.
1839
- **Location**: Use this field to configure the location of BigQuery datasets as per [the BigQuery documentation](https://cloud.google.com/bigquery/docs/locations).
19-
- **Workload Identity Federation**: Support for Workload Identity Federation with BigQuery service accounts is coming soon
2040

2141
<IpAllowlist />

‎docs/snippets/dwh/bigquery/cloud_wif.mdx‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
### Create a Workload Identity pool and provider:
2+
3+
1. In the Cloud Console, go to: IAM & Admin > Workload Identity Federation. Click 'CREATE POOL' (or select an existing pool), give it a name (e.g. `elementary-identity-pool`), and continue.
4+
5+
2. On the pool, click 'ADD A PROVIDER' and choose 'AWS'. Set the provider name to `Elementary AWS` and the AWS account ID to `743289191656` (Elementary's AWS account), then continue.
6+
7+
<img
8+
src="/pics/bigquery-wif/add_aws_provider.png"
9+
alt="Add AWS provider"
10+
/>
11+
12+
3. In the provider's attribute conditions, add the following condition. This is the Elementary role for GCP — only Elementary can authenticate through this provider. Then save the provider:
13+
14+
```
15+
attribute.aws_role == "arn:aws:sts::743289191656:assumed-role/elementary-gcp-wif-role-prod-elementary-cloud"
16+
```
17+
18+
<img
19+
src="/pics/bigquery-wif/attribute_condition.png"
20+
alt="Attribute condition"
21+
/>
22+
23+
### Grant BigQuery permissions to the identity pool principal:
24+
25+
4. Go to IAM & Admin > IAM and click 'GRANT ACCESS'. In 'New principals', paste the following principalSet. Replace `PROJECT_NUMBER` with your GCP project number and `POOL_NAME` with the pool you created above:
26+
27+
```
28+
principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/attribute.aws_role/arn:aws:sts::743289191656:assumed-role/elementary-gcp-wif-role-prod-elementary-cloud
29+
```
30+
31+
5. Assign the following project-level roles to the principalSet and save. You will grant `BigQuery Data Viewer` at the dataset level in the next step:
32+
- **BigQuery Job User**
33+
- **BigQuery Metadata Viewer**
34+
- **BigQuery Resource Viewer**
35+
36+
### Grant principal access to specific datasets:
37+
38+
6. In the [BigQuery console](https://console.cloud.google.com/bigquery), find your Elementary dataset in the 'Explorer' tab. Click the three dots next to it > Share > ADD PRINCIPAL. Paste the same principalSet from step 4, select `BigQuery Data Viewer`, and save. If your dbt dataset is in a different project, also grant it `BigQuery Metadata Viewer` and `BigQuery Resource Viewer`.
39+
40+
### Download the credential configuration file:
41+
42+
7. Back in IAM & Admin > Workload Identity Federation, select your pool and click 'GRANT ACCESS' at the top.
43+
44+
8. Choose 'Grant access using federated identities (Recommended)' and click 'DOWNLOAD CONFIG'.
45+
46+
<img
47+
src="/pics/bigquery-wif/grant_access_federated_identities.png"
48+
alt="Grant access using federated identities"
49+
/>
50+
51+
9. In the 'Configure your application' panel, select the provider you created (e.g. `Elementary AWS`) from the Provider dropdown, then click 'DOWNLOAD CONFIG' to download the JSON configuration file.
52+
53+
<img
54+
src="/pics/bigquery-wif/configure_application_download_config.png"
55+
alt="Select provider and download config"
56+
/>
57+
58+
10. Upload this JSON file to Elementary Cloud in the connection form below.

0 commit comments

Comments
 (0)