A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Traditionally, applications running outside Google Cloud can use static service account keys to access Google Cloud resources. In Vector, most GCP related sinks (in particular for us the 'gcp_stackdriver_logs'), use a 'credentials_path' config where we provide the path to the 'service_account' json file, containing the static keys. However, service account keys are powerful credentials and can present a security risk if they are not managed correctly.

Workload identity enables you to assign distinct, fine-grained identities and authorization. Workload identity is the recommended way for applications to access AWS and Google Cloud services.

Customer's want to be able to authenticate GCP sinks using service account impersonation (aka 'external_accounts'), rather than long-lived credentials, in the same way AWS sinks have the ability to authenticate using a 'role_arn' and 'web_identity_token'.

https://cloud.google.com/iam/docs/workload-identity-federation

Attempted Solutions

Current blocker is a generic error: "Invalid GCP Credentials", when using a type: "external_account" for the credentials file in vector.

Proposal

Without Workload Identity, the type of the credentials file is service_account. These credentials include a private RSA key, in the private_key field, to be able to authenticate to gcp. This private key needs to be kept secure and is most often not rotated.

{
   "type": "service_account",
   "project_id": "test-project",
   "private_key_id": "123absc5678993dabd942adf0ff0812c789f",
   "private_key": "<private_key>",
   "client_email": "test-email@test-project.iam.gserviceaccount.com",
   "client_id": "1006844567890123456789",
   "auth_uri": "https://accounts.google.com/o/oauth2/auth",
   "token_uri": "https://oauth2.googleapis.com/token",
   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test-email@test-project.iam.gserviceaccount.com"
}

With Workload Identity, the type of the credentials is external_account, and the audience is the target audience which is the workload identity provider. The service_account_impersonation_url key contains the resource url of the service account that can be impersonated with these credentials. credentials_source.file is the path to the oidc token, which is exchanged for a google access token. The oidc token is then rotated every hour and thus credentials are short lived.

{
   "type": "external_account",
   "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider",
   "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
   "token_url": "https://sts.googleapis.com/v1/token",
   "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/test-service-account@test-project.iam.gserviceaccount.com:generateAccessToken",
   "credential_source": {
      "file": "/path/to/oidc/token",
      "format": {
         "type": "text"
      }
   }
}

References

Recent GCP work for another related type of auth "GKE workload identity":

• Use GCP instance service account for sinks #2243

Version

0.26.0