diff --git a/docs/cloud/integrations/log-streaming/gcs.mdx b/docs/cloud/integrations/log-streaming/gcs.mdx index 39c6a82fb..3512585f5 100644 --- a/docs/cloud/integrations/log-streaming/gcs.mdx +++ b/docs/cloud/integrations/log-streaming/gcs.mdx @@ -22,12 +22,40 @@ Before configuring log streaming to GCS, you'll need: - The bucket must exist and be accessible - You'll need the bucket path (e.g., `gs://my-logs-bucket`) -2. **Google Cloud Service Account** - A service account with permissions to write to the bucket - - Required role: `Storage Object User` (roles/storage.objectUser) - - You'll need to generate a service account JSON key file - - The service account key file must be uploaded in Elementary - - **Workload Identity Federation**: Support for Workload Identity Federation with BigQuery service accounts is coming soon +2. **Authentication** - Either a Google Cloud service account or a Workload Identity Federation setup, with the **Storage Object User** (`roles/storage.objectUser`) role granted on the bucket. See [Authentication methods](#authentication-methods) below for both options. +## Authentication methods + +Elementary supports two authentication methods for GCS. Pick the one that fits your security model: + +- **Service account** — create a service account, download its JSON key, and upload the key to Elementary. Simplest to set up. +- **Workload Identity Federation (WIF)** — Elementary authenticates from its AWS role through a federated identity. No long-lived credentials are stored in Elementary. + +Select a tab below and follow the steps for your chosen method. + + + + 1. Go to [Google Cloud Console > IAM & Admin > Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) and create a service account (or select an existing one). + + 2. Grant the service account the **Storage Object User** (`roles/storage.objectUser`) role on your GCS bucket. + + 3. Generate a JSON key for the service account: + 1. Select your service account. + 2. Click the three dots menu and select 'Manage keys'. + 3. Click 'ADD KEY' and select 'Create new key'. + 4. Choose 'JSON' format and click 'CREATE'. The JSON file downloads automatically. + + 4. You will upload this JSON key to Elementary in the connection form below, in the **Service account file** field. + + + The WIF setup is identical to the one documented for BigQuery — the only GCS-specific change is the permission you grant to the pool principal. Follow the Workload Identity Federation steps on the [BigQuery connection page](/cloud/integrations/dwh/bigquery) (select the **Workload Identity Federation** tab), with the following change: + + - In the step where you grant IAM roles to the principalSet, **do not** grant the BigQuery roles. Instead, grant the **Storage Object User** (`roles/storage.objectUser`) role on your GCS bucket. You can do this from the bucket's 'Permissions' tab: 'GRANT ACCESS' → paste the same principalSet → assign `Storage Object User`. + - You can skip the dataset-level grant step entirely — it is BigQuery-specific. + + Once you have the downloaded WIF credential configuration file, upload it to Elementary in the connection form below, in the **WIF credential file** field. + + ## Configuring Log Streaming to GCS @@ -42,14 +70,8 @@ Before configuring log streaming to GCS, you'll need: 4. Enter your GCS configuration: - **Bucket Path**: The full GCS bucket path (e.g., `gs://my-logs-bucket`) - - **Service Account Key File**: Upload your Google Cloud service account JSON key file - - To generate a service account key file: - 1. Go to [Google Cloud Console > IAM & Admin > Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) - 2. Select your service account (or create a new one) - 3. Click the three dots menu and select "Manage keys" - 4. Click "ADD KEY" and select "Create new key" - 5. Choose "JSON" format and click "CREATE" - 6. The JSON file will be downloaded automatically + - **Authentication method**: Use the toggle to select **Service account** or **Workload Identity Federation**, matching the method you set up in [Authentication methods](#authentication-methods) above. + - **Service account file** (Service account method) or **WIF credential file** (Workload Identity Federation method): Upload the JSON file you prepared. 5. Click **Save** to enable log streaming diff --git a/docs/pics/bigquery-wif/add_aws_provider.png b/docs/pics/bigquery-wif/add_aws_provider.png new file mode 100644 index 000000000..fad8dfbd2 Binary files /dev/null and b/docs/pics/bigquery-wif/add_aws_provider.png differ diff --git a/docs/pics/bigquery-wif/attribute_condition.png b/docs/pics/bigquery-wif/attribute_condition.png new file mode 100644 index 000000000..80bfe9ec0 Binary files /dev/null and b/docs/pics/bigquery-wif/attribute_condition.png differ diff --git a/docs/pics/bigquery-wif/configure_application_download_config.png b/docs/pics/bigquery-wif/configure_application_download_config.png new file mode 100644 index 000000000..d941562aa Binary files /dev/null and b/docs/pics/bigquery-wif/configure_application_download_config.png differ diff --git a/docs/pics/bigquery-wif/grant_access_federated_identities.png b/docs/pics/bigquery-wif/grant_access_federated_identities.png new file mode 100644 index 000000000..fd5a9e7f6 Binary files /dev/null and b/docs/pics/bigquery-wif/grant_access_federated_identities.png differ diff --git a/docs/snippets/cloud/integrations/bigquery.mdx b/docs/snippets/cloud/integrations/bigquery.mdx index c69b55239..3be5e6381 100644 --- a/docs/snippets/cloud/integrations/bigquery.mdx +++ b/docs/snippets/cloud/integrations/bigquery.mdx @@ -1,21 +1,41 @@ import CloudServiceAccount from '/snippets/dwh/bigquery/cloud_service_account.mdx'; +import CloudWIF from '/snippets/dwh/bigquery/cloud_wif.mdx'; import PermissionsAndSecurity from '/snippets/cloud/integrations/permissions-and-security.mdx'; import IpAllowlist from '/snippets/cloud/integrations/ip-allowlist.mdx'; This guide contains the necessary steps to connect a BigQuery environment to your Elementary account. - +## Choose an authentication method + +Elementary supports two authentication methods for BigQuery. Pick the one that fits your security model: + +- **Service account** — create a service account, download its JSON key, and upload the key to Elementary. Simplest to set up. +- **Workload Identity Federation (WIF)** — Elementary authenticates from its AWS role through a federated identity. No long-lived credentials are stored in Elementary. + +Select a tab below and follow the steps for your chosen method. + + + + + + + + + ### Fill the connection form -Provide the following fields: +Use the **Authentication method** toggle at the top of the form to select either **Service account** or **Workload Identity Federation**, matching the method you set up above. The credentials upload field changes based on your selection: + +- **Service account file** (Service account method): the service account JSON key file you downloaded. +- **WIF credential file** (Workload Identity Federation method): the external account JSON configuration file you downloaded from the identity pool. + +Then provide the remaining fields: -- **Service account file**: The service account file you generated for Elementary. For more information, refer to [Create BigQuery service account](./bigquery#create-bigquery-service-account). - **Project**: The name of your BigQuery project. - **Elementary dataset**: The name of your Elementary dataset. Usually `[dataset name]_elementary`. - **Location**: Use this field to configure the location of BigQuery datasets as per [the BigQuery documentation](https://cloud.google.com/bigquery/docs/locations). -- **Workload Identity Federation**: Support for Workload Identity Federation with BigQuery service accounts is coming soon diff --git a/docs/snippets/dwh/bigquery/cloud_wif.mdx b/docs/snippets/dwh/bigquery/cloud_wif.mdx new file mode 100644 index 000000000..0045f673d --- /dev/null +++ b/docs/snippets/dwh/bigquery/cloud_wif.mdx @@ -0,0 +1,58 @@ +### Create a Workload Identity pool and provider: + +1. In the Cloud Console, go to: IAM & Admin > Workload Identity Federation. Click 'CREATE POOL' (or select an existing pool), give it a name (e.g. `elementary-identity-pool`), and continue. + +2. On the pool, click 'ADD A PROVIDER' and choose 'AWS'. Set the provider name to `Elementary AWS` and the AWS account ID to `743289191656` (Elementary's AWS account), then continue. + +Add AWS provider + +3. In the provider's attribute conditions, add the following condition. This is the Elementary role for GCP — only Elementary can authenticate through this provider. Then save the provider: + + ``` + attribute.aws_role == "arn:aws:sts::743289191656:assumed-role/elementary-gcp-wif-role-prod-elementary-cloud" + ``` + +Attribute condition + +### Grant BigQuery permissions to the identity pool principal: + +4. Go to IAM & Admin > IAM and click 'GRANT ACCESS'. In 'New principals', paste the following principalSet. Replace `PROJECT_NUMBER` with your GCP project number and `POOL_NAME` with the pool you created above: + + ``` + principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/attribute.aws_role/arn:aws:sts::743289191656:assumed-role/elementary-gcp-wif-role-prod-elementary-cloud + ``` + +5. Assign the following project-level roles to the principalSet and save. You will grant `BigQuery Data Viewer` at the dataset level in the next step: + - **BigQuery Job User** + - **BigQuery Metadata Viewer** + - **BigQuery Resource Viewer** + +### Grant principal access to specific datasets: + +6. In the [BigQuery console](https://console.cloud.google.com/bigquery), find your Elementary dataset in the 'Explorer' tab. Click the three dots next to it > Share > ADD PRINCIPAL. Paste the same principalSet from step 4, select `BigQuery Data Viewer`, and save. If your dbt dataset is in a different project, also grant it `BigQuery Metadata Viewer` and `BigQuery Resource Viewer`. + +### Download the credential configuration file: + +7. Back in IAM & Admin > Workload Identity Federation, select your pool and click 'GRANT ACCESS' at the top. + +8. Choose 'Grant access using federated identities (Recommended)' and click 'DOWNLOAD CONFIG'. + +Grant access using federated identities + +9. In the 'Configure your application' panel, select the provider you created (e.g. `Elementary AWS`) from the Provider dropdown, then click 'DOWNLOAD CONFIG' to download the JSON configuration file. + +Select provider and download config + +10. Upload this JSON file to Elementary Cloud in the connection form below.