Pod Identity Association also subject to cache race condition?

[joshfrench]

What happened:
Similar to #174 but specific to pod identity associations, we're observing the expected AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE env var is absent when a service account and pod are created within a short window. Typically we'll experience something like this:

• Programmatically create a pod identity association, service account annotated with IAM role, and pod in short succession
• The pod comes up, but AWS operations error with An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://oidc.eks...
• Examining the pod env, note that AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE is missing but AWS_WEB_IDENTITY_TOKEN_FILE is set
• Restart pod
• Note that AWS_WEB_IDENTITY_TOKEN_FILE is now replaced with AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE and pod operates as expected.

What you expected to happen:
If all the prerequisites are satisfied, pods should get the correct pod identity association mutation regardless of timing.

How to reproduce it (as minimally and precisely as possible):

1. Create an IAM role with the correct pod identity association trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:TagSession",
                "sts:AssumeRole"
            ],
        }
    ]
}

2. Create an EKS cluster, enabling the EKS Pod Identity Agent add-on.
3. Run aws eks update-kubeconfig --name my-cluster
4. Run:

$ aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace default \
  --service-account test-sa \
  --role-arn arn:aws:iam::111111111111:role/test-role && \
sleep 0.75s && \
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: test-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/test-role
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: default
spec:
  serviceAccountName: test-sa
  containers:
    - name: test
      image: amazon/aws-cli:latest
      imagePullPolicy: IfNotPresent
      command:
        - aws
        - sts
        - get-caller-identity
EOF

When waiting ~750ms or less between creating the association and submitting the SA, I consistently get the incorrect AWS_WEB_IDENTITY_TOKEN_FILE. Above ~1s seems to be reliably sufficient to get the correct mutation.

Anything else we need to know?:
I'm wondering if something like #236 and/or #252 should be applied to the FileConfig, to allow the cache some time to catch up or to provide a fallback in case of cache miss. The scenario in which a serviceaccount and a pod are created in a short timeframe is common with CI/CD and infrastructure-as-code.

Environment:

• AWS Region: us-east-2
• EKS Platform version: eks.6
• Kubernetes version: 1.32
• Webhook Version: ¯\_(ツ)_/¯ whatever EKS is running under the hood

[haoranleo]

Hi @joshfrench this is because of the propagation delay for the pod identity associations.

Basically what happens under the hood is that after user creates the pod identity association through AWS API aws eks create-pod-identity-association, the pod identity association is created in EKS control plane when the API returns 200. Then there is a async workflow propagating the pod identity association to the cluster. The pod-identity-webhook on the cluster wouldn't be able to mutate the pod with AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE environment variable until the pod identity association is propagated to the cluster.

We realize there is a friction when using EKS Pod Identity. It's a known issue and it is being tracked internally. Before the issue is fully fixed we advise users to wait for sometime (<5s would be enough) after the pod identity association is created, before using that pod identity association in the cluster.

[johan-vandeweerd-hs]

@haoranleo is there a Github issue of some sort that we can subscribe on to track the progress of this?

We use Argocd to deploy our applications and AWS Controller for Kubernetes to create the pod identity association the first time an application is deployed and we run into this issue as well.

We are currently playing with a workaround where we add a Kubernetes Job to our Argocd Application that runs with a pre-sync hook. The job checks the necessary env variables and see if they are set or not. If not, the job fails and retries. This is used to block before the deployment gets synced without using a fixed time to wait. It works but won't win any beauty awards.

---
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app: my-app
  name: my-app
  annotations:
    helm.sh/hook-weight: "50"
spec:
  template:
    spec:
      serviceAccountName: my-app
      restartPolicy: Never
      containers:
        - name: my-app
          image: public.ecr.aws/ubuntu/ubuntu:24.10
          command:
            - sh
            - "-c"
            - "env | grep AWS_CONTAINER ; if [ $? -ne 0 ]; then echo 'No env set'; exit 1; else echo 'Found'; fi"
  backoffLimit: 4
  activeDeadlineSeconds: 60
  ttlSecondsAfterFinished: 60
---
apiVersion: eks.services.k8s.aws/v1alpha1
kind: PodIdentityAssociation
metadata:
  annotations:
    helm.sh/hook: pre-install
    helm.sh/hook-weight: "-10"
  labels:
    app: my-app
  name: my-app
spec:
  clusterName: <CLUSTER_NAME>
  namespace: default
  roleARN: <ROLE_ARN>
  serviceAccount: my-app

[csantanapr]

A better alternative than using a job, is to inject into the pod the environment variables and volumemount that the webhook would inject, and not depend on the webhook.

I wrote a blog post here with more details of the solution https://builder.aws.com/content/32Qdx3gzUqBnPvJOl9bbS8DuRgN/deterministic-eks-pod-identity-with-kro-and-ack

[johan-vandeweerd-hs]

Interesting. Only two (minor) issues I currently see are:

1. Tight coupling with pod identity "internals" like the env variables and volume mount
2. Debugging is a bit harder in case there is a pod identity misconfiguration (incorrect trust policy on the role or missing pod identity association). In the standard scenario there wouldn't be any env vars injected which gives a clou to how to troubleshoot the issue

Both not the end of the world and might outweight the downsides of the job. Will have a more in-depth look at this. Thanks for the suggestion!

[lostick]

A better alternative than using a job, is to inject into the pod the environment variables and volumemount that the webhook would inject, and not depend on the webhook.

Not quite sure this is an improvement over the pre-sync job IMO. Having to inject these env vars ends up being more fiddly.