🤖 feat: migrate sandbox EKS Terraform to EKS Auto Mode

[ThomasK33]

Summary

Migrate the Terraform-managed sandbox-eks cluster from a managed node group setup to EKS Auto Mode.

Background

The previous Terraform configuration used an EKS managed node group and did not model Auto Mode-managed compute/storage/load balancing behavior. This change aligns the sandbox cluster with EKS Auto Mode and removes now-obsolete node group configuration.

Implementation

• Added EKS Auto Mode IAM policy attachments to the cluster IAM role.
• Added a dedicated Auto Mode node IAM role with minimal worker + ECR pull policies.
• Enabled EKS Auto Mode in aws_eks_cluster:
  • compute_config (general-purpose node pool)
  • kubernetes_network_config.elastic_load_balancing
  • storage_config.block_storage
• Removed the managed node group resources.
• Updated addon dependencies to depend on cluster readiness.
• Removed stale node group outputs and variables.
• Updated Terraform README docs and estimated-cost guidance for Auto Mode.

Validation

• make verify-vendor
• make test
• make build
• make lint
• terraform -chdir=terraform fmt -check -diff
• terraform -chdir=terraform init -backend=false
• terraform -chdir=terraform validate

Risks

• In-place migration may cause workload rescheduling during apply.
• Existing node-group-specific automation depending on removed outputs/variables will need to be updated.

---

📋 Implementation Plan

Plan: Migrate Terraform-managed sandbox EKS to EKS Auto Mode

Context / Why

The current Terraform in terraform/ provisions a standard EKS cluster with a managed node group and only the core EKS managed add-ons (coredns, kube-proxy, vpc-cni). It does not deploy the EBS CSI driver, so dynamic provisioning of EBS-backed PersistentVolumes via PVCs is not guaranteed.

Goal: migrate the existing sandbox-eks cluster to EKS Auto Mode so AWS manages:

• Compute scaling (managed instances)
• Block storage (EBS CSI)
• Elastic Load Balancing integration

This plan assumes the following (based on your answers):

• In-place migration (no blue/green cluster)
• PVC/PV data does not need to be preserved (sandbox)
• No SSH/SSM access to nodes is OK

Evidence

• Current Terraform cluster + node group + add-ons: terraform/eks.tf
• IAM roles/policies: terraform/iam.tf
• AWS provider version pinned: terraform/.terraform.lock.hcl (hashicorp/aws 6.31.0)
• AWS provider supports EKS Auto Mode via aws_eks_cluster.compute_config, kubernetes_network_config.elastic_load_balancing, and storage_config.block_storage: hashicorp/aws provider docs for aws_eks_cluster (v6.31.0)
• AWS EKS Auto Mode overview and managed components: AWS EKS User Guide (automode.html)

---

Migration plan (recommended: staged applies)

Phase 0 — Pre-flight safety checks

1. Confirm you can access current cluster (so you can validate after each apply):

   • terraform -chdir=terraform output -raw kubeconfig_command (then run it)
   • kubectl get nodes -o wide
   • kubectl get pods -A

2. Capture a quick baseline (helpful for rollback/debug):

   • terraform -chdir=terraform state list
   • terraform -chdir=terraform plan (save the output)

3. Communicate expected disruption

   • Enabling Auto Mode and later removing the managed node group can cause short-lived scheduling disruption.

Why staged applies?

Auto Mode enablement is a control-plane update plus new IAM requirements. Keeping the managed node group initially provides a safety net while you validate Auto Mode health (and avoids a “no nodes in cluster” scenario if something is misconfigured).

---

Phase 1 — Enable Auto Mode capabilities (keep existing node group)

1) Terraform changes (IAM)

Edit terraform/iam.tf:

1. Keep existing cluster role aws_iam_role.eks_cluster but attach the additional AWS-managed policies required for Auto Mode:

   • arn:aws:iam::aws:policy/AmazonEKSComputePolicy
   • arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy
   • arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy
   • arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy

2. Add a dedicated node role for Auto Mode managed instances (don’t reuse the managed node group role so it can be deleted later):

   • Trust principal: ec2.amazonaws.com
   • Attach:
     • arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy
     • arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly

Shape of the intended additions:

# terraform/iam.tf

resource "aws_iam_role" "auto_mode_node" {
  name = "${var.cluster_name}-auto-node-role"
  assume_role_policy = data.aws_iam_policy_document.node_group_assume_role.json

  tags = {
    Name = "${var.cluster_name}-auto-node-role"
  }
}

resource "aws_iam_role_policy_attachment" "auto_mode_node_worker_minimal" {
  role       = aws_iam_role.auto_mode_node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy"
}

resource "aws_iam_role_policy_attachment" "auto_mode_node_ecr_pull_only" {
  role       = aws_iam_role.auto_mode_node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly"
}

resource "aws_iam_role_policy_attachment" "eks_cluster_compute_policy" {
  role       = aws_iam_role.eks_cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSComputePolicy"
}

resource "aws_iam_role_policy_attachment" "eks_cluster_block_storage_policy" {
  role       = aws_iam_role.eks_cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy"
}

resource "aws_iam_role_policy_attachment" "eks_cluster_load_balancing_policy" {
  role       = aws_iam_role.eks_cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy"
}

resource "aws_iam_role_policy_attachment" "eks_cluster_networking_policy" {
  role       = aws_iam_role.eks_cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy"
}

2) Terraform changes (EKS cluster Auto Mode config)

Edit terraform/eks.tf in aws_eks_cluster.this:

Add all three of the following blocks (they must be enabled together):

resource "aws_eks_cluster" "this" {
  ...
  bootstrap_self_managed_addons = false

  compute_config {
    enabled       = true
    node_pools    = ["general-purpose"]
    node_role_arn = aws_iam_role.auto_mode_node.arn
  }

  kubernetes_network_config {
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }

  ...
}

Also update the cluster depends_on to include the new policy attachments so deletion ordering remains correct:

depends_on = [
  aws_iam_role_policy_attachment.eks_cluster_policy,
  aws_iam_role_policy_attachment.eks_cluster_compute_policy,
  aws_iam_role_policy_attachment.eks_cluster_block_storage_policy,
  aws_iam_role_policy_attachment.eks_cluster_load_balancing_policy,
  aws_iam_role_policy_attachment.eks_cluster_networking_policy,
]

3) Apply (Phase 1)

Run:

• terraform -chdir=terraform fmt
• terraform -chdir=terraform init -upgrade -backend-config=backend.hcl
• terraform -chdir=terraform plan
• terraform -chdir=terraform apply

4) Validate (Phase 1)

1. Confirm Auto Mode flags are enabled:

   • aws eks describe-cluster --region $(terraform -chdir=terraform output -raw region) --name $(terraform -chdir=terraform output -raw cluster_name)

2. Confirm nodes still exist (node group should still be present):

   • kubectl get nodes -o wide

3. Confirm EBS CSI is available and storage works:

   • kubectl get csidrivers
   • kubectl get storageclass
   • Create a tiny test PVC + pod and ensure it binds/mounts.

---

Phase 2 — Retire the managed node group (move fully to Auto Mode)

1) Create Auto Mode capacity (if needed)

EKS Auto Mode generally provisions nodes based on unschedulable pods. To force a clean cutover:

• Cordon + drain one managed node group node and watch for replacement capacity:
  • kubectl cordon <node>
  • kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
  • kubectl get nodes -w

If the cluster ends up briefly without schedulable nodes, this is acceptable for sandbox, but you should still watch kube-system pods recover.

2) Terraform changes (remove node group)

Once you’ve confirmed Auto Mode is healthy:

1. Remove aws_eks_node_group.this from terraform/eks.tf.
2. Remove the node-group IAM role and its attachments from terraform/iam.tf:
   • aws_iam_role.node_group
   • aws_iam_role_policy_attachment.node_group_*
3. Update the EKS add-ons’ depends_on in terraform/eks.tf:
   • Change from depends_on = [aws_eks_node_group.this] to depends_on = [aws_eks_cluster.this] (or remove depends_on entirely if not needed).
4. Update outputs:
   • Remove or replace output "node_group_name" in terraform/outputs.tf.

3) Apply (Phase 2)

• terraform -chdir=terraform plan
• terraform -chdir=terraform apply

4) Validate (Phase 2)

• kubectl get nodes -o wide (should show only Auto Mode-managed nodes)
• kubectl get pods -A (all Running/Ready)
• Re-run the PVC provisioning test

---

Cleanup / follow-ups

• Update terraform/README.md to reflect:

  • No managed node group
  • Auto Mode managed components (compute + storage + ELB)
  • Updated cost estimate (Auto Mode adds an additional management fee)

• Optional: remove now-unused variables from terraform/variables.tf (node_*) after confirming no external automation relies on them.

---

Rollback strategy (sandbox-friendly)

• If Phase 1 fails: revert the Auto Mode blocks and IAM policy attachments, keep the managed node group, and re-apply.
• If Phase 2 fails (node group removed and Auto Mode not provisioning): re-introduce aws_eks_node_group and apply, or destroy/recreate the sandbox cluster.

Because PVC data preservation is not required, the simplest escape hatch is terraform -chdir=terraform destroy followed by apply once the configuration is corrected.

---

Generated with mux • Model: openai:gpt-5.3-codex • Thinking: xhigh • Cost: $0.38

[ThomasK33]

@codex review

Please review this Terraform migration of sandbox-eks to EKS Auto Mode.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. You're on a roll.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Follow-up commit pushed after live apply testing to satisfy EKS Auto Mode prerequisites:

• access_config.authentication_mode = API_AND_CONFIG_MAP
• compute_config.node_pools = ["general-purpose", "system"]
• cluster role trust includes sts:TagSession

Local validation rerun:

• terraform fmt/validate
• make verify-vendor
• make test
• make build
• make lint

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Keep it up!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".