Get error during creation of pod with service-account that has eks.amazonaws.com/role-arn annotation

[cynepco3hahue]

What happened:

One of our pods should assume a specific AWS role and we are configuring it via service account annotation. After eks cluster upgrade the mutation of the pod stopped working and it had only a default node group role.
I changed the mutatingwebhookconfigurations policy from Ignore to Fail for pod-identity-webhook under the cluster and after that pod failed to start with the error

Error creating: Internal error occurred: failed calling webhook "iam-for-pods.amazonaws.com": received invalid webhook response: expected webhook response of admission.k8s.io/v1, Kind=AdmissionReview, got /, Kind=

After some additional investigation, I found that the problem solved once I change

- admissionReviewVersions:
  - v1

to

- admissionReviewVersions:
  - v1beta1

IIUC the problem that mutation webhook handler is using deprecated version of admission API v1beta1 when it should use v1

What you expected to happen:
The mutation of the pod should work without any errors.

How to reproduce it (as minimally and precisely as possible):

1. Get the EKS cluster
2. Try to configure the pod with some role, you can follow up https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

Anything else we need to know?:

Environment:

• AWS Region: us-east-1
• EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.15
• Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.27
• Webhook Version: Unsure how can I check it

[OscarPoels]

Got the exact same problem on a EKS cluster upgrade, not possible to use the v1 API version when it should

[Andrei-Vasil]

Same issue here, do you have any follow up on this?

[dom-colangelo]

I hit this same issue on a non-EKS cluster and traced it to the same root cause. The handler builds the response AdmissionReview without copying TypeMeta from the request, so the response serializes with empty apiVersion/kind — which Kubernetes rejects.

I've opened a fix in #304 that echoes the request TypeMeta back in the response and replaces the decoder with json.Unmarshal so v1 requests parse correctly. All existing tests pass and a regression test for the v1 round-trip is included.