🤖 feat: migrate sandbox EKS Terraform to EKS Auto Mode (#39)

ThomasK33 · web-flow · commit 5c418a7ee06b · 2026-02-10T18:38:21.000Z
## Summary
Migrate the Terraform-managed `sandbox-eks` cluster from a managed node
group setup to EKS Auto Mode.

## Background
The previous Terraform configuration used an EKS managed node group and
did not model Auto Mode-managed compute/storage/load balancing behavior.
This change aligns the sandbox cluster with EKS Auto Mode and removes
now-obsolete node group configuration.

## Implementation
- Added EKS Auto Mode IAM policy attachments to the cluster IAM role.
- Added a dedicated Auto Mode node IAM role with minimal worker + ECR
pull policies.
- Enabled EKS Auto Mode in `aws_eks_cluster`:
  - `compute_config` (`general-purpose` node pool)
  - `kubernetes_network_config.elastic_load_balancing`
  - `storage_config.block_storage`
- Removed the managed node group resources.
- Updated addon dependencies to depend on cluster readiness.
- Removed stale node group outputs and variables.
- Updated Terraform README docs and estimated-cost guidance for Auto
Mode.

## Validation
- `make verify-vendor`
- `make test`
- `make build`
- `make lint`
- `terraform -chdir=terraform fmt -check -diff`
- `terraform -chdir=terraform init -backend=false`
- `terraform -chdir=terraform validate`

## Risks
- In-place migration may cause workload rescheduling during apply.
- Existing node-group-specific automation depending on removed
outputs/variables will need to be updated.

---

&lt;details&gt;
&lt;summary&gt;📋 Implementation Plan&lt;/summary&gt;

# Plan: Migrate Terraform-managed sandbox EKS to EKS Auto Mode

## Context / Why
The current Terraform in `terraform/` provisions a standard EKS cluster
with a managed node group and only the core EKS managed add-ons
(`coredns`, `kube-proxy`, `vpc-cni`). It does **not** deploy the EBS CSI
driver, so dynamic provisioning of EBS-backed PersistentVolumes via PVCs
is not guaranteed.

Goal: migrate the existing **`sandbox-eks`** cluster to **EKS Auto
Mode** so AWS manages:
- Compute scaling (managed instances)
- Block storage (EBS CSI)
- Elastic Load Balancing integration

This plan assumes the following (based on your answers):
- **In-place migration** (no blue/green cluster)
- **PVC/PV data does not need to be preserved** (sandbox)
- **No SSH/SSM access to nodes is OK**

## Evidence
- Current Terraform cluster + node group + add-ons: `terraform/eks.tf`
- IAM roles/policies: `terraform/iam.tf`
- AWS provider version pinned: `terraform/.terraform.lock.hcl`
(hashicorp/aws **6.31.0**)
- AWS provider supports EKS Auto Mode via
`aws_eks_cluster.compute_config`,
`kubernetes_network_config.elastic_load_balancing`, and
`storage_config.block_storage`: hashicorp/aws provider docs for
`aws_eks_cluster` (v6.31.0)
- AWS EKS Auto Mode overview and managed components: AWS EKS User Guide
(`automode.html`)

---

## Migration plan (recommended: staged applies)

### Phase 0 — Pre-flight safety checks
1. **Confirm you can access current cluster** (so you can validate after
each apply):
- `terraform -chdir=terraform output -raw kubeconfig_command` (then run
it)
   - `kubectl get nodes -o wide`
   - `kubectl get pods -A`

2. **Capture a quick baseline** (helpful for rollback/debug):
   - `terraform -chdir=terraform state list`
   - `terraform -chdir=terraform plan` (save the output)

3. **Communicate expected disruption**
- Enabling Auto Mode and later removing the managed node group can cause
short-lived scheduling disruption.

&lt;details&gt;
&lt;summary&gt;Why staged applies?&lt;/summary&gt;

Auto Mode enablement is a control-plane update plus new IAM
requirements. Keeping the managed node group initially provides a safety
net while you validate Auto Mode health (and avoids a “no nodes in
cluster” scenario if something is misconfigured).

&lt;/details&gt;

---

### Phase 1 — Enable Auto Mode capabilities (keep existing node group)

#### 1) Terraform changes (IAM)
Edit `terraform/iam.tf`:

1. **Keep** existing cluster role `aws_iam_role.eks_cluster` but attach
the additional AWS-managed policies required for Auto Mode:
   - `arn:aws:iam::aws:policy/AmazonEKSComputePolicy`
   - `arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy`
   - `arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy`
   - `arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy`

2. **Add** a dedicated node role for Auto Mode managed instances (don’t
reuse the managed node group role so it can be deleted later):
   - Trust principal: `ec2.amazonaws.com`
   - Attach:
     - `arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy`
     - `arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly`

Shape of the intended additions:

```hcl
# terraform/iam.tf

resource "aws_iam_role" "auto_mode_node" {
  name = "${var.cluster_name}-auto-node-role"
  assume_role_policy = data.aws_iam_policy_document.node_group_assume_role.json

  tags = {
    Name = "${var.cluster_name}-auto-node-role"
  }
}

resource "aws_iam_role_policy_attachment" "auto_mode_node_worker_minimal" {
  role       = aws_iam_role.auto_mode_node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy"
}

resource "aws_iam_role_policy_attachment" "auto_mode_node_ecr_pull_only" {
  role       = aws_iam_role.auto_mode_node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly"
}

resource "aws_iam_role_policy_attachment" "eks_cluster_compute_policy" {
  role       = aws_iam_role.eks_cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSComputePolicy"
}

resource "aws_iam_role_policy_attachment" "eks_cluster_block_storage_policy" {
  role       = aws_iam_role.eks_cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy"
}

resource "aws_iam_role_policy_attachment" "eks_cluster_load_balancing_policy" {
  role       = aws_iam_role.eks_cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy"
}

resource "aws_iam_role_policy_attachment" "eks_cluster_networking_policy" {
  role       = aws_iam_role.eks_cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy"
}
```

#### 2) Terraform changes (EKS cluster Auto Mode config)
Edit `terraform/eks.tf` in `aws_eks_cluster.this`:

Add **all three** of the following blocks (they must be enabled
together):

```hcl
resource "aws_eks_cluster" "this" {
  ...
  bootstrap_self_managed_addons = false

  compute_config {
    enabled       = true
    node_pools    = ["general-purpose"]
    node_role_arn = aws_iam_role.auto_mode_node.arn
  }

  kubernetes_network_config {
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }

  ...
}
```

Also update the cluster `depends_on` to include the new policy
attachments so deletion ordering remains correct:

```hcl
depends_on = [
  aws_iam_role_policy_attachment.eks_cluster_policy,
  aws_iam_role_policy_attachment.eks_cluster_compute_policy,
  aws_iam_role_policy_attachment.eks_cluster_block_storage_policy,
  aws_iam_role_policy_attachment.eks_cluster_load_balancing_policy,
  aws_iam_role_policy_attachment.eks_cluster_networking_policy,
]
```

#### 3) Apply (Phase 1)
Run:
- `terraform -chdir=terraform fmt`
- `terraform -chdir=terraform init -upgrade -backend-config=backend.hcl`
- `terraform -chdir=terraform plan`
- `terraform -chdir=terraform apply`

#### 4) Validate (Phase 1)
1. Confirm Auto Mode flags are enabled:
- `aws eks describe-cluster --region $(terraform -chdir=terraform output
-raw region) --name $(terraform -chdir=terraform output -raw
cluster_name)`

2. Confirm nodes still exist (node group should still be present):
   - `kubectl get nodes -o wide`

3. Confirm EBS CSI is available and storage works:
   - `kubectl get csidrivers`
   - `kubectl get storageclass`
   - Create a tiny test PVC + pod and ensure it binds/mounts.

---

### Phase 2 — Retire the managed node group (move fully to Auto Mode)

#### 1) Create Auto Mode capacity (if needed)
EKS Auto Mode generally provisions nodes based on unschedulable pods. To
force a clean cutover:
- Cordon + drain one managed node group node and watch for replacement
capacity:
  - `kubectl cordon &lt;node&gt;`
  - `kubectl drain &lt;node&gt; --ignore-daemonsets --delete-emptydir-data`
  - `kubectl get nodes -w`

If the cluster ends up briefly without schedulable nodes, this is
acceptable for sandbox, but you should still watch `kube-system` pods
recover.

#### 2) Terraform changes (remove node group)
Once you’ve confirmed Auto Mode is healthy:

1. Remove `aws_eks_node_group.this` from `terraform/eks.tf`.
2. Remove the node-group IAM role and its attachments from
`terraform/iam.tf`:
   - `aws_iam_role.node_group`
   - `aws_iam_role_policy_attachment.node_group_*`
3. Update the EKS add-ons’ `depends_on` in `terraform/eks.tf`:
- Change from `depends_on = [aws_eks_node_group.this]` to `depends_on =
[aws_eks_cluster.this]` (or remove `depends_on` entirely if not needed).
4. Update outputs:
- Remove or replace `output "node_group_name"` in
`terraform/outputs.tf`.

#### 3) Apply (Phase 2)
- `terraform -chdir=terraform plan`
- `terraform -chdir=terraform apply`

#### 4) Validate (Phase 2)
- `kubectl get nodes -o wide` (should show only Auto Mode-managed nodes)
- `kubectl get pods -A` (all Running/Ready)
- Re-run the PVC provisioning test

---

## Cleanup / follow-ups
- Update `terraform/README.md` to reflect:
  - No managed node group
  - Auto Mode managed components (compute + storage + ELB)
  - Updated cost estimate (Auto Mode adds an additional management fee)

- Optional: remove now-unused variables from `terraform/variables.tf`
(`node_*`) after confirming no external automation relies on them.

---

## Rollback strategy (sandbox-friendly)
- If Phase 1 fails: revert the Auto Mode blocks and IAM policy
attachments, keep the managed node group, and re-apply.
- If Phase 2 fails (node group removed and Auto Mode not provisioning):
re-introduce `aws_eks_node_group` and apply, or destroy/recreate the
sandbox cluster.

Because PVC data preservation is not required, the simplest escape hatch
is `terraform -chdir=terraform destroy` followed by `apply` once the
configuration is corrected.

&lt;/details&gt;

---

_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking:
`xhigh` • Cost: `$0.38`_

&lt;!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh
costs=0.38 --&gt;
diff --git a/terraform/README.md b/terraform/README.md
@@ -9,13 +9,12 @@ This directory provisions a cost-optimized Amazon EKS sandbox cluster in region
   - 2 private subnets across the first two standard availability zones in the selected region
   - Internet Gateway
   - Single NAT Gateway (lower cost than one per AZ)
-- IAM roles for EKS control plane and worker nodes
+- IAM roles for the EKS control plane and EKS Auto Mode managed instances
 - EKS cluster (`sandbox-eks`, Kubernetes `1.35`) with public and private API endpoint access
-- One managed node group:
-  - Instance type: `t3.medium`
-  - Desired/min/max size: `2/1/3`
-  - Disk size: `20 GiB`
-  - AMI type: `AL2023_x86_64_STANDARD`
+- EKS Auto Mode enabled for:
+  - Compute (`general-purpose` node pool)
+  - Elastic Load Balancing integration
+  - Block storage integration (EBS CSI managed by EKS)
 - EKS managed add-ons: `coredns`, `kube-proxy`, `vpc-cni`
 - Remote Terraform state in AWS S3 with lockfile-based locking (no DynamoDB)
 
@@ -164,11 +163,11 @@ aws eks update-kubeconfig --region eu-central-1 --name sandbox-eks
 ## Estimated cost (rough)
 
 - EKS control plane: **~$0.10/hour**
-- 2x `t3.medium` worker nodes: **~$0.08/hour**
-- 1x NAT Gateway: **~$0.045/hour**
-- **Total: ~ $0.225/hour (~$5.40/day)**
+- EKS Auto Mode: additional per-instance management fee (see current EKS pricing for your region)
+- EC2 and EBS usage: varies with actual workload and Auto Mode scaling behavior
+- 1x NAT Gateway: **~$0.045/hour** (plus data processing)
 
-> Note: Data transfer and NAT data processing charges are additional.
+> Note: Data transfer, NAT data processing, and workload-driven compute/storage usage are additional.
 
 ## Cleanup
 
diff --git a/terraform/eks.tf b/terraform/eks.tf
@@ -5,35 +5,42 @@ resource "aws_eks_cluster" "this" {
 
   bootstrap_self_managed_addons = false
 
-  vpc_config {
-    subnet_ids              = aws_subnet.private[*].id
-    endpoint_public_access  = true
-    endpoint_private_access = true
+  access_config {
+    authentication_mode = "API_AND_CONFIG_MAP"
   }
 
-  depends_on = [aws_iam_role_policy_attachment.eks_cluster_policy]
-}
+  compute_config {
+    enabled       = true
+    node_pools    = ["general-purpose", "system"]
+    node_role_arn = aws_iam_role.auto_mode_node.arn
+  }
 
-resource "aws_eks_node_group" "this" {
-  cluster_name    = aws_eks_cluster.this.name
-  node_group_name = "${var.cluster_name}-nodes"
-  node_role_arn   = aws_iam_role.node_group.arn
-  subnet_ids      = aws_subnet.private[*].id
+  kubernetes_network_config {
+    elastic_load_balancing {
+      enabled = true
+    }
+  }
 
-  instance_types = var.node_instance_types
-  ami_type       = "AL2023_x86_64_STANDARD"
-  disk_size      = var.node_disk_size
+  storage_config {
+    block_storage {
+      enabled = true
+    }
+  }
 
-  scaling_config {
-    desired_size = var.node_desired_size
-    min_size     = var.node_min_size
-    max_size     = var.node_max_size
+  vpc_config {
+    subnet_ids              = aws_subnet.private[*].id
+    endpoint_public_access  = true
+    endpoint_private_access = true
   }
 
   depends_on = [
-    aws_iam_role_policy_attachment.node_group_worker_policy,
-    aws_iam_role_policy_attachment.node_group_cni_policy,
-    aws_iam_role_policy_attachment.node_group_ecr_readonly,
+    aws_iam_role_policy_attachment.auto_mode_node_worker_minimal,
+    aws_iam_role_policy_attachment.auto_mode_node_ecr_pull_only,
+    aws_iam_role_policy_attachment.eks_cluster_policy,
+    aws_iam_role_policy_attachment.eks_cluster_compute_policy,
+    aws_iam_role_policy_attachment.eks_cluster_block_storage_policy,
+    aws_iam_role_policy_attachment.eks_cluster_load_balancing_policy,
+    aws_iam_role_policy_attachment.eks_cluster_networking_policy,
   ]
 }
 
@@ -43,21 +50,21 @@ resource "aws_eks_addon" "coredns" {
   addon_name                  = "coredns"
   resolve_conflicts_on_create = "OVERWRITE"
 
-  depends_on = [aws_eks_node_group.this]
+  depends_on = [aws_eks_cluster.this]
 }
 
 resource "aws_eks_addon" "kube_proxy" {
   cluster_name                = aws_eks_cluster.this.name
   addon_name                  = "kube-proxy"
   resolve_conflicts_on_create = "OVERWRITE"
 
-  depends_on = [aws_eks_node_group.this]
+  depends_on = [aws_eks_cluster.this]
 }
 
 resource "aws_eks_addon" "vpc_cni" {
   cluster_name                = aws_eks_cluster.this.name
   addon_name                  = "vpc-cni"
   resolve_conflicts_on_create = "OVERWRITE"
 
-  depends_on = [aws_eks_node_group.this]
+  depends_on = [aws_eks_cluster.this]
 }
diff --git a/terraform/iam.tf b/terraform/iam.tf
@@ -1,6 +1,6 @@
 data "aws_iam_policy_document" "eks_cluster_assume_role" {
   statement {
-    actions = ["sts:AssumeRole"]
+    actions = ["sts:AssumeRole", "sts:TagSession"]
 
     principals {
       type        = "Service"
@@ -23,7 +23,27 @@ resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
   policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
 }
 
-data "aws_iam_policy_document" "node_group_assume_role" {
+resource "aws_iam_role_policy_attachment" "eks_cluster_compute_policy" {
+  role       = aws_iam_role.eks_cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSComputePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "eks_cluster_block_storage_policy" {
+  role       = aws_iam_role.eks_cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "eks_cluster_load_balancing_policy" {
+  role       = aws_iam_role.eks_cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "eks_cluster_networking_policy" {
+  role       = aws_iam_role.eks_cluster.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy"
+}
+
+data "aws_iam_policy_document" "auto_mode_node_assume_role" {
   statement {
     actions = ["sts:AssumeRole"]
 
@@ -34,26 +54,21 @@ data "aws_iam_policy_document" "node_group_assume_role" {
   }
 }
 
-resource "aws_iam_role" "node_group" {
-  name               = "${var.cluster_name}-node-role"
-  assume_role_policy = data.aws_iam_policy_document.node_group_assume_role.json
+resource "aws_iam_role" "auto_mode_node" {
+  name               = "${var.cluster_name}-auto-node-role"
+  assume_role_policy = data.aws_iam_policy_document.auto_mode_node_assume_role.json
 
   tags = {
-    Name = "${var.cluster_name}-node-role"
+    Name = "${var.cluster_name}-auto-node-role"
   }
 }
 
-resource "aws_iam_role_policy_attachment" "node_group_worker_policy" {
-  role       = aws_iam_role.node_group.name
-  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
-}
-
-resource "aws_iam_role_policy_attachment" "node_group_cni_policy" {
-  role       = aws_iam_role.node_group.name
-  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+resource "aws_iam_role_policy_attachment" "auto_mode_node_worker_minimal" {
+  role       = aws_iam_role.auto_mode_node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy"
 }
 
-resource "aws_iam_role_policy_attachment" "node_group_ecr_readonly" {
-  role       = aws_iam_role.node_group.name
-  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+resource "aws_iam_role_policy_attachment" "auto_mode_node_ecr_pull_only" {
+  role       = aws_iam_role.auto_mode_node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly"
 }
diff --git a/terraform/outputs.tf b/terraform/outputs.tf
@@ -19,11 +19,6 @@ output "cluster_security_group_id" {
   value       = aws_eks_cluster.this.vpc_config[0].cluster_security_group_id
 }
 
-output "node_group_name" {
-  description = "Name of the managed EKS node group."
-  value       = aws_eks_node_group.this.node_group_name
-}
-
 output "region" {
   description = "AWS region where the cluster is deployed."
   value       = var.aws_region
diff --git a/terraform/variables.tf b/terraform/variables.tf
@@ -35,32 +35,3 @@ variable "vpc_cidr" {
   }
 }
 
-variable "node_instance_types" {
-  description = "EC2 instance types for the managed node group."
-  type        = list(string)
-  default     = ["t3.medium"]
-}
-
-variable "node_desired_size" {
-  description = "Desired number of worker nodes in the managed node group."
-  type        = number
-  default     = 2
-}
-
-variable "node_min_size" {
-  description = "Minimum number of worker nodes in the managed node group."
-  type        = number
-  default     = 1
-}
-
-variable "node_max_size" {
-  description = "Maximum number of worker nodes in the managed node group."
-  type        = number
-  default     = 3
-}
-
-variable "node_disk_size" {
-  description = "Disk size in GiB for each node in the managed node group."
-  type        = number
-  default     = 20
-}
