Skip to content

Commit ecc7e33

Browse files
committed
feat!(workload-identity): replace kubectl-wrapper with native kubernetes_annotations
Replace the legacy `module "annotate-sa"` (which executed `kubectl_wrapper.sh` via local-exec) in `modules/workload-identity` with the native `resource "kubernetes_annotations" "main"` from `hashicorp/kubernetes`. This resolves Python `AttributeError: 'Credentials' object has no attribute 'private_key_id'` failures in `gke-gcloud-auth-plugin` / `gcloud config config-helper` when executing inside keyless authentication pipelines (Service Account Impersonation, Workload Identity Federation, ADC). Key changes: - Replace `kubectl-wrapper` local-exec bash execution with declarative `kubernetes_annotations` using Server-Side Apply (`force = true`). - Remove obsolete input variables (`cluster_name`, `location`, `impersonate_service_account`, `use_existing_context`). - Ensure `google_service_account_iam_member.main` and `var.module_depends_on` are explicitly included in `depends_on` for both KSA creation and annotation resources to prevent IAM propagation race conditions. - Update module metadata (`metadata.yaml`, `metadata.display.yaml`), README, and examples. - Add upgrade guide `docs/upgrading_to_v45.0.md` detailing breaking changes, provider prerequisites, and state migration.
1 parent bb176b2 commit ecc7e33

8 files changed

Lines changed: 115 additions & 77 deletions

File tree

docs/upgrading_to_v45.0.md

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Upgrading to v45.0
2+
3+
The v45.0 release of *terraform-google-kubernetes-engine* contains breaking interface changes in the `workload-identity` submodule.
4+
5+
## Workload Identity Submodule (`modules/workload-identity`)
6+
7+
### 1. Native `kubernetes_annotations` Replacement
8+
9+
The `workload-identity` submodule no longer uses `kubectl-wrapper` (`local-exec` bash script execution) to annotate existing Kubernetes Service Accounts. Instead, it uses the native [`kubernetes_annotations`](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/annotations) resource from the `hashicorp/kubernetes` provider.
10+
11+
#### Kubernetes Provider Prerequisite
12+
Because `kubernetes_annotations` is a native Terraform provider resource, callers **must** have an initialized and authenticated `kubernetes` provider configured in their root module context:
13+
14+
```hcl
15+
provider "kubernetes" {
16+
host = "https://${module.gke.endpoint}"
17+
token = data.google_client_config.default.access_token
18+
cluster_ca_certificate = base64decode(module.gke.ca_certificate)
19+
}
20+
```
21+
22+
> **Note:** If you previously configured `ignore_annotations` in your `kubernetes` provider block to ignore Workload Identity annotations (e.g., `ignore_annotations = ["^iam.gke.io\\/.*"]`) to prevent conflicts with the legacy `kubectl-wrapper`, you must remove that configuration. Otherwise, the new native `kubernetes_annotations` resource will not function correctly.
23+
24+
---
25+
26+
### 2. Removed Input Variables
27+
28+
The following input variables were previously used to configure `gcloud container clusters get-credentials` and `kubectl` inside `kubectl_wrapper.sh`. They have been removed as they are no longer required by the module:
29+
30+
- `cluster_name`
31+
- `location`
32+
- `impersonate_service_account`
33+
- `use_existing_context`
34+
35+
#### Expected Error if Unchanged
36+
If you attempt to apply module calls containing these deleted inputs, Terraform will return an error during configuration validation:
37+
38+
```text
39+
Error: Unsupported argument
40+
41+
on main.tf line 32, in module "workload_identity":
42+
32: cluster_name = module.gke.name
43+
44+
An argument named "cluster_name" is not expected here.
45+
```
46+
47+
#### Migration Example
48+
49+
Update your `module "workload_identity"` calls to remove the deleted arguments:
50+
51+
```diff
52+
module "workload_identity" {
53+
source = "terraform-google-modules/kubernetes-engine/google//modules/workload-identity"
54+
- version = "~> 44.0"
55+
+ version = "~> 45.0"
56+
57+
project_id = var.project_id
58+
name = "my-app"
59+
- cluster_name = module.gke.name
60+
- location = module.gke.location
61+
namespace = "default"
62+
use_existing_k8s_sa = true
63+
k8s_sa_name = "existing-ksa"
64+
}
65+
```
66+
67+
---
68+
69+
### 3. State Migration Strategy
70+
71+
For existing deployments where `use_existing_k8s_sa = true` and `annotate_k8s_sa = true`:
72+
73+
1. Terraform will plan to destroy the legacy `module.annotate-sa` resources and create `kubernetes_annotations.main[0]`.
74+
2. `kubernetes_annotations.main[0]` uses `force = true` and Server-Side Apply to adopt the existing `"iam.gke.io/gcp-service-account"` annotation seamlessly.
75+
3. **Proactive State Cleanup for CI/CD Runners:** Standard CI/CD runner environments (Terraform Cloud, Spacelift, Atlantis, minimal GitHub Actions runners) typically lack local `gcloud` and `kubectl` binaries. Attempting to run `terraform apply` directly may fail during the destroy phase of `module.annotate-sa`.
76+
77+
To prevent build failures, remove the legacy module state **prior to running `terraform apply`**:
78+
79+
```bash
80+
# Standard module call
81+
terraform state rm 'module.workload_identity.module.annotate-sa'
82+
83+
# Module call using count or for_each
84+
terraform state rm 'module.workload_identity["app"].module.annotate-sa'
85+
terraform state rm 'module.workload_identity[0].module.annotate-sa'
86+
```
87+
88+
*Note: Deployments created with `use_existing_k8s_sa = false` (default) do not use `module.annotate-sa` and require no state migration.*

examples/acm-terraform-blog-part3/terraform/gke.tf

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,7 @@ module "wi" {
5050
source = "terraform-google-modules/kubernetes-engine/google//modules/workload-identity"
5151
version = "~> 44.0"
5252
gcp_sa_name = "cnrmsa"
53-
cluster_name = module.gke.name
5453
name = "cnrm-controller-manager"
55-
location = var.zone
5654
use_existing_k8s_sa = true
5755
annotate_k8s_sa = false
5856
namespace = "cnrm-system"

examples/workload_identity/main.tf

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,6 @@ module "workload_identity_existing_ksa" {
7979

8080
project_id = var.project_id
8181
name = "existing-${module.gke.name}"
82-
cluster_name = module.gke.name
83-
location = module.gke.location
8482
namespace = "default"
8583
use_existing_k8s_sa = true
8684
k8s_sa_name = kubernetes_service_account_v1.test.metadata[0].name

modules/workload-identity/README.md

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,6 @@ resource "kubernetes_service_account_v1" "preexisting" {
8383
module "my-app-workload-identity" {
8484
source = "terraform-google-modules/kubernetes-engine/google//modules/workload-identity"
8585
use_existing_k8s_sa = true
86-
cluster_name = "my-k8s-cluster-name"
87-
location = "my-k8s-cluster-location"
8886
name = kubernetes_service_account_v1.preexisting.metadata[0].name
8987
namespace = kubernetes_service_account_v1.preexisting.metadata[0].namespace
9088
project_id = var.project_id
@@ -147,22 +145,18 @@ Error: Get "http://localhost/api/v1/namespaces/default/serviceaccounts/your-serv
147145
| additional\_projects | A list of roles to be added to the created service account for additional projects | `map(list(string))` | `{}` | no |
148146
| annotate\_k8s\_sa | Annotate the kubernetes service account with 'iam.gke.io/gcp-service-account' annotation. Valid in cases when an existing SA is used. | `bool` | `true` | no |
149147
| automount\_service\_account\_token | Enable automatic mounting of the service account token | `bool` | `false` | no |
150-
| cluster\_name | Cluster name. Required if using existing KSA. | `string` | `""` | no |
151148
| gcp\_sa\_create\_ignore\_already\_exists | If set to true, skip service account creation if a service account with the same email already exists. | `bool` | `null` | no |
152149
| gcp\_sa\_description | The Service Google service account desciption; if null, will be left out | `string` | `null` | no |
153150
| gcp\_sa\_display\_name | The Google service account display name; if null, a default string will be used | `string` | `null` | no |
154151
| gcp\_sa\_name | Name for the Google service account; overrides `var.name`. | `string` | `null` | no |
155152
| image\_pull\_secrets | A list of references to secrets in the same namespace to use for pulling any images in pods that reference this Service Account | `list(string)` | `[]` | no |
156-
| impersonate\_service\_account | An optional service account to impersonate for gcloud commands. If this service account is not specified, the module will use Application Default Credentials. | `string` | `""` | no |
157-
| k8s\_sa\_name | Name for the Kubernetes service account; overrides `var.name`. `cluster_name` and `location` must be set when this input is specified. | `string` | `null` | no |
153+
| k8s\_sa\_name | Name for the Kubernetes service account; overrides `var.name`. | `string` | `null` | no |
158154
| k8s\_sa\_project\_id | GCP project ID of the k8s service account; overrides `var.project_id`. | `string` | `null` | no |
159-
| location | Cluster location (region if regional cluster, zone if zonal cluster). Required if using existing KSA. | `string` | `""` | no |
160155
| module\_depends\_on | List of modules or resources to depend on before annotating KSA. If multiple, all items must be the same type. | `list(any)` | `[]` | no |
161156
| name | Name for both service accounts. The GCP SA will be truncated to the first 30 chars if necessary. | `string` | n/a | yes |
162157
| namespace | Namespace for the Kubernetes service account | `string` | `"default"` | no |
163158
| project\_id | GCP project ID | `string` | n/a | yes |
164159
| roles | A list of roles to be added to the created service account | `list(string)` | `[]` | no |
165-
| use\_existing\_context | An optional flag to use local kubectl config context. | `bool` | `false` | no |
166160
| use\_existing\_gcp\_sa | Use an existing Google service account instead of creating one | `bool` | `false` | no |
167161
| use\_existing\_k8s\_sa | Use an existing kubernetes service account instead of creating one | `bool` | `false` | no |
168162

modules/workload-identity/main.tf

Lines changed: 24 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ locals {
2727
output_k8s_namespace = var.use_existing_k8s_sa ? var.namespace : kubernetes_service_account_v1.main[0].metadata[0].namespace
2828

2929
k8s_sa_project_id = var.k8s_sa_project_id != null ? var.k8s_sa_project_id : var.project_id
30-
k8s_sa_gcp_derived_name = "serviceAccount:${local.k8s_sa_project_id}.svc.id.goog[${var.namespace}/${local.output_k8s_name}]"
30+
k8s_sa_gcp_derived_name = "serviceAccount:${local.k8s_sa_project_id}.svc.id.goog[${var.namespace}/${local.k8s_given_name}]"
3131

3232
sa_binding_additional_project = distinct(flatten([for project, roles in var.additional_projects : [for role in roles : { project_id = project, role_name = role }]]))
3333
}
@@ -74,24 +74,34 @@ resource "kubernetes_service_account_v1" "main" {
7474
"iam.gke.io/gcp-service-account" = local.gcp_sa_email
7575
}
7676
}
77+
78+
depends_on = [
79+
google_service_account_iam_member.main,
80+
var.module_depends_on,
81+
]
7782
}
7883

79-
module "annotate-sa" {
80-
source = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
81-
version = "~> 4.0"
84+
resource "kubernetes_annotations" "main" {
85+
count = var.use_existing_k8s_sa && var.annotate_k8s_sa ? 1 : 0
86+
87+
api_version = "v1"
88+
kind = "ServiceAccount"
8289

83-
enabled = var.use_existing_k8s_sa && var.annotate_k8s_sa
84-
skip_download = true
85-
cluster_name = var.cluster_name
86-
cluster_location = var.location
87-
project_id = local.k8s_sa_project_id
88-
impersonate_service_account = var.impersonate_service_account
89-
use_existing_context = var.use_existing_context
90+
metadata {
91+
name = local.k8s_given_name
92+
namespace = local.output_k8s_namespace
93+
}
94+
95+
annotations = {
96+
"iam.gke.io/gcp-service-account" = local.gcp_sa_email
97+
}
9098

91-
kubectl_create_command = "kubectl annotate --overwrite sa -n ${local.output_k8s_namespace} ${local.k8s_given_name} iam.gke.io/gcp-service-account=${local.gcp_sa_email}"
92-
kubectl_destroy_command = "kubectl annotate sa -n ${local.output_k8s_namespace} ${local.k8s_given_name} iam.gke.io/gcp-service-account-"
99+
force = true
93100

94-
module_depends_on = var.module_depends_on
101+
depends_on = [
102+
google_service_account_iam_member.main,
103+
var.module_depends_on,
104+
]
95105
}
96106

97107
resource "google_service_account_iam_member" "main" {

modules/workload-identity/metadata.display.yaml

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -37,9 +37,6 @@ spec:
3737
automount_service_account_token:
3838
name: automount_service_account_token
3939
title: Automount Service Account Token
40-
cluster_name:
41-
name: cluster_name
42-
title: Cluster Name
4340
gcp_sa_create_ignore_already_exists:
4441
name: gcp_sa_create_ignore_already_exists
4542
title: Gcp Sa Create Ignore Already Exists
@@ -55,18 +52,12 @@ spec:
5552
image_pull_secrets:
5653
name: image_pull_secrets
5754
title: Image Pull Secrets
58-
impersonate_service_account:
59-
name: impersonate_service_account
60-
title: Impersonate Service Account
6155
k8s_sa_name:
6256
name: k8s_sa_name
6357
title: K8s Sa Name
6458
k8s_sa_project_id:
6559
name: k8s_sa_project_id
6660
title: K8s Sa Project Id
67-
location:
68-
name: location
69-
title: Location
7061
module_depends_on:
7162
name: module_depends_on
7263
title: Module Depends On
@@ -82,9 +73,6 @@ spec:
8273
roles:
8374
name: roles
8475
title: Roles
85-
use_existing_context:
86-
name: use_existing_context
87-
title: Use Existing Context
8876
use_existing_gcp_sa:
8977
name: use_existing_gcp_sa
9078
title: Use Existing Gcp Sa

modules/workload-identity/metadata.yaml

Lines changed: 1 addition & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -145,16 +145,8 @@ spec:
145145
description: Use an existing Google service account instead of creating one
146146
varType: bool
147147
defaultValue: false
148-
- name: cluster_name
149-
description: Cluster name. Required if using existing KSA.
150-
varType: string
151-
defaultValue: ""
152-
- name: location
153-
description: Cluster location (region if regional cluster, zone if zonal cluster). Required if using existing KSA.
154-
varType: string
155-
defaultValue: ""
156148
- name: k8s_sa_name
157-
description: Name for the Kubernetes service account; overrides `var.name`. `cluster_name` and `location` must be set when this input is specified.
149+
description: Name for the Kubernetes service account; overrides `var.name`.
158150
varType: string
159151
- name: k8s_sa_project_id
160152
description: GCP project ID of the k8s service account; overrides `var.project_id`.
@@ -183,14 +175,6 @@ spec:
183175
description: A list of roles to be added to the created service account
184176
varType: list(string)
185177
defaultValue: []
186-
- name: impersonate_service_account
187-
description: An optional service account to impersonate for gcloud commands. If this service account is not specified, the module will use Application Default Credentials.
188-
varType: string
189-
defaultValue: ""
190-
- name: use_existing_context
191-
description: An optional flag to use local kubectl config context.
192-
varType: bool
193-
defaultValue: false
194178
- name: module_depends_on
195179
description: List of modules or resources to depend on before annotating KSA. If multiple, all items must be the same type.
196180
varType: list(any)

modules/workload-identity/variables.tf

Lines changed: 1 addition & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -36,20 +36,9 @@ variable "use_existing_gcp_sa" {
3636
default = false
3737
}
3838

39-
variable "cluster_name" {
40-
description = "Cluster name. Required if using existing KSA."
41-
type = string
42-
default = ""
43-
}
44-
45-
variable "location" {
46-
description = "Cluster location (region if regional cluster, zone if zonal cluster). Required if using existing KSA."
47-
type = string
48-
default = ""
49-
}
5039

5140
variable "k8s_sa_name" {
52-
description = "Name for the Kubernetes service account; overrides `var.name`. `cluster_name` and `location` must be set when this input is specified."
41+
description = "Name for the Kubernetes service account; overrides `var.name`."
5342
type = string
5443
default = null
5544
}
@@ -96,17 +85,6 @@ variable "roles" {
9685
default = []
9786
}
9887

99-
variable "impersonate_service_account" {
100-
description = "An optional service account to impersonate for gcloud commands. If this service account is not specified, the module will use Application Default Credentials."
101-
type = string
102-
default = ""
103-
}
104-
105-
variable "use_existing_context" {
106-
description = "An optional flag to use local kubectl config context."
107-
type = bool
108-
default = false
109-
}
11088

11189
variable "module_depends_on" {
11290
description = "List of modules or resources to depend on before annotating KSA. If multiple, all items must be the same type."

0 commit comments

Comments
 (0)