kubernetes_cluster: concurrent provider exec corrupts auth binary install (exit 126)

[sanmesh-kakade]

The kubernetes_provider_exec script in the EKS/GKE cluster modules installs its auth binary on first use:

command -v aws-iam-authenticator || (curl -sLo /tmp/aws-iam-authenticator ... && chmod +x ... && mv /tmp/aws-iam-authenticator /usr/local/bin/...)

All kubernetes-family providers (kubernetes, helm, kubectl, kubernetes-alpha) run this exec concurrently at plan start. They all download to the same fixed /tmp path, so parallel invocations overwrite each other mid-download and the losing provider executes a half-written binary:

Error: ... getting credentials: exec: executable bash failed with exit code 126

Reproduced by running the exact script 4x in parallel in a release pod: 2 succeeded, 2 exited 126. The failure is consistent in real releases because the collision happens on every plan.

Affected:

• modules/kubernetes_cluster/eks_standard/1.0/outputs.tf
• modules/kubernetes_cluster/eks_automode/1.0/outputs.tf
• modules/kubernetes_cluster/gke/1.0/outputs.tf (same pattern with gke-auth-plugin, plus a tar extract to another fixed /tmp path)

Fix (already applied to the tekion fork, eks_standard_tekion in squad-3, verified working):

command -v aws-iam-authenticator >/dev/null 2>&1 || flock /usr/local/bin/.aws-iam-authenticator.lock bash -c 'command -v aws-iam-authenticator >/dev/null 2>&1 && exit 0; t=$(mktemp /usr/local/bin/.aws-iam-authenticator.XXXXXX) && curl -fsSLo $t <url> && chmod +x $t && mv -f $t /usr/local/bin/aws-iam-authenticator'; aws-iam-authenticator token ...

• flock + re-check serializes to a single download
• mktemp inside /usr/local/bin keeps the temp file on the same filesystem, so mv -f is an atomic rename — a concurrent exec never sees a partial binary (/tmp can be a separate mount, where mv degrades to a non-atomic copy)
• curl -f stops an HTTP error page from being installed as the binary

Longer term it's worth baking these binaries into the runner image so the download path never runs.