🤖 chore: move terraform state to S3 with lockfile locking

[ThomasK33]

Summary

Move the terraform/ root module from local state to AWS S3 remote state with native lockfile-based locking (no DynamoDB). Bootstrap the S3 bucket, import all 25 existing AWS resources into the new remote state, and align the Terraform config with the deployed infrastructure.

Background

The terraform/ directory provisions a sandbox EKS cluster in eu-central-1. It was using the default local backend, meaning state lived wherever terraform apply was last run. This is fragile for team use and risks state loss.

Implementation

S3 backend configuration

• terraform/versions.tf: Added backend "s3" block with use_lockfile = true, encrypt = true, region eu-central-1. Bucket/key are supplied at init time via -backend-config (not hardcoded).
• Bumped required_version from >= 1.5 to >= 1.11 (minimum for S3 lockfile support).

S3 bucket bootstrap (out-of-band)

• Created bucket coder-k8s-tfstate-112158171837 in eu-central-1 with versioning, default AES256 encryption, and all public access blocked.

Resource import

• Discovered all 25 existing AWS resources (VPC, subnets, IGW, NAT GW, EIP, route tables, RT associations, IAM roles, policy attachments, EKS cluster, node group, 3 addons).
• Used import {} blocks to import everything in a single terraform apply.
• terraform plan is a clean no-op after import.

Config drift fixes

• terraform/vpc.tf: Replaced cidrsubnets() (which generated /20 blocks) with cidrsubnet() per-subnet (generating /24 blocks matching deployed infra). CIDRs are still derived from var.vpc_cidr.
• terraform/eks.tf: Added bootstrap_self_managed_addons = false to match the deployed cluster.

Repo housekeeping

• .gitignore: Added standard Terraform ignores (state files, .terraform/, crash logs, plan files, override files, terraform/backend.hcl). Does not ignore .terraform.lock.hcl.
• flake.nix: Added terraform to the Nix dev shell, with allowUnfreePredicate scoped to just terraform (BSL 1.1 license).

Documentation

• terraform/README.md: Updated to reflect S3 remote state, Terraform >= 1.11, added "Remote state backend" section with bucket bootstrap, IAM, auth, and migration docs. Updated Usage section with -backend-config workflow.

Validation

• terraform plan → "No changes. Infrastructure matches the configuration."
• State file confirmed in S3 (50KB, AES256 encrypted)
• make build ✅
• make test ✅
• make lint ✅
• nix-instantiate --parse flake.nix ✅
• terraform fmt -check ✅

Risks

• Low: The cidrsubnet() change produces identical CIDRs to deployed infra, confirmed by clean plan. If var.vpc_cidr were ever changed, subnet CIDRs would shift — but that's a VPC rebuild regardless.
• Low: BSL license allowance in flake is scoped to terraform only.

---

📋 Implementation Plan

Plan: Move terraform/ state to AWS S3 (S3 lockfile, no DynamoDB)

Context / Why

We have a Terraform root module under terraform/ that provisions a sandbox EKS cluster in eu-central-1. Today it uses the default local backend (state lives on whoever ran terraform apply).

Goal:

• Store Terraform state in AWS S3 (in eu-central-1).
• Use S3-native state locking (Terraform’s use_lockfile), no DynamoDB.
• Add repo housekeeping so state/working dirs don’t get committed.
• Ensure the Nix flake dev shell includes the Terraform CLI.
• After the S3 bucket exists, discover any already-existing AWS resources for this stack and import them into the new remote state, so we don’t “orphan” resources.

Evidence (what I checked)

• terraform/versions.tf: no backend configured; required_version = ">= 1.5".
• terraform/*.tf: root module resources are all plain aws_* resources (no external modules), so imports are addressable directly (e.g., aws_vpc.this, aws_eks_cluster.this).
• .gitignore: currently no Terraform ignores.
• flake.nix: dev shell includes awscli2 but does not include terraform.
• Terraform docs (v1.11.0) for S3 backend show:
  • Locking via use_lockfile = true.
  • DynamoDB locking is deprecated and will be removed in a future minor version.
  • Lock file uses <key>.tflock in the same bucket/prefix.
• AWS S3 docs indicate strong read-after-write consistency and atomic updates per key, but S3 itself does not provide a built-in “object locking for concurrent writers” mechanism, so Terraform’s lockfile is still required.

---

Implementation Plan

1) Repo changes (backend config, docs, housekeeping, Nix)

1.1. Update Terraform version + S3 backend config

File: terraform/versions.tf

• Bump Terraform minimum version so the repo’s documented expectations match S3 lockfile support.
  • Recommend: required_version = ">= 1.11" (lockfile is no longer described as experimental; DynamoDB locking is deprecated).
• Add an S3 backend block with lockfile enabled and encryption enabled.

There are two workable patterns; pick one:

Option A (recommended): keep bucket/key out of git; commit only the backend type + safety settings

• Commit:
  • backend type s3
  • use_lockfile = true
  • encrypt = true
  • region = "eu-central-1"
• Provide bucket + key via -backend-config at init time (see §2).

terraform {
  required_version = ">= 1.11"

  backend "s3" {
    region       = "eu-central-1"
    encrypt      = true
    use_lockfile = true
    # bucket/key supplied via -backend-config to avoid hardcoding a globally-unique name
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

Option B: commit the full backend config (bucket + key) into the repo

• Only do this if there is exactly one shared AWS account/environment for this Terraform root module.
• Still set use_lockfile = true and encrypt = true.

Plan assumes Option A to avoid bikeshedding the globally unique bucket name in git.

1.2. Document remote state + locking + bootstrap steps

File: terraform/README.md

Update:

• Prereqs: Terraform >= 1.11.
• Replace “Local Terraform state” note with “Remote state in S3”.
• Add a “Remote state backend (S3 + lockfile)” section including:
  • Bucket bootstrap commands (create bucket, enable versioning, block public access, enable default encryption).
  • The backend config approach (Option A) and the terraform init command to supply bucket/key.
  • AWS auth note: the S3 backend authenticates separately from the AWS provider. If you’re using TF_VAR_aws_profile, also set AWS_PROFILE (or set profile = ... in the backend config file) so terraform init can access the S3 bucket.
  • Explain lockfile behavior: lock stored at <key>.tflock; why s3:DeleteObject is needed for the lock object.

1.3. Housekeeping: ignore local Terraform artifacts (but commit dependency lock file)

File: .gitignore

Add standard Terraform ignores:

• State files:
  • *.tfstate
  • *.tfstate.*
• Working directories:
  • .terraform/
• Crash logs:
  • crash.log
  • crash.*.log
• Plan files:
  • *.tfplan
  • *.tfplan.*
• Override files:
  • override.tf, override.tf.json, *_override.tf, *_override.tf.json
• CLI config files:
  • .terraformrc, terraform.rc
• Local backend config (for terraform init -backend-config=...; do not commit):
  • terraform/backend.hcl

Important nuance:

• Do not ignore .terraform.lock.hcl (provider dependency lock). Best practice is to commit it.

Suggested block:

# Terraform
*.tfstate
*.tfstate.*
.terraform/
crash.log
crash.*.log
*.tfplan
*.tfplan.*
override.tf
override.tf.json
*_override.tf
*_override.tf.json
.terraformrc
terraform.rc

# Backend config (local-only)
terraform/backend.hcl

1.4. Add Terraform CLI to the Nix flake dev shell

File: flake.nix

Add terraform to devShells.default.packages near awscli2:

# Cloud tooling
awscli2
terraform

Follow-up (verification step):

• Run nix develop and confirm terraform version is >= 1.11. If nixpkgs’ terraform is older for any reason, switch to a versioned package (e.g., terraform_1 / terraform_1_11 depending on nixpkgs naming).

---

2) Bootstrap the S3 backend bucket (eu-central-1)

Terraform can’t create its own backend bucket (chicken-and-egg), so create it out-of-band.

2.1 Choose names

• Bucket name: must be globally unique. Recommended pattern: coder-k8s-tfstate-<account-id> or coder-k8s-tfstate-<org>-<env>.
• State key: keep it stable and descriptive, e.g. terraform-ncp3/sandbox-eks/terraform.tfstate.

2.2 Create + harden the bucket

Example AWS CLI (adjust bucket name):

REGION=eu-central-1
BUCKET=<globally-unique-bucket>

aws s3api create-bucket \
  --bucket "$BUCKET" \
  --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION"

# Strongly recommended for state recovery
aws s3api put-bucket-versioning \
  --bucket "$BUCKET" \
  --versioning-configuration Status=Enabled

# Block all public access
aws s3api put-public-access-block \
  --bucket "$BUCKET" \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# Default encryption (SSE-S3). If you prefer SSE-KMS, use KMS instead.
aws s3api put-bucket-encryption \
  --bucket "$BUCKET" \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}
    }]
  }'

Optional but recommended hardening:

• Bucket policy enforcing TLS (aws:SecureTransport) and encryption headers.
• Object Ownership = Bucket owner enforced (disables ACLs).

2.3 IAM permissions for Terraform backend access

Whoever runs Terraform needs:

• Read/write the state object: s3:GetObject, s3:PutObject.
• Read/write/delete the lock object <key>.tflock: s3:GetObject, s3:PutObject, s3:DeleteObject.
• List the bucket (ideally prefix-restricted): s3:ListBucket.

Add/update an IAM policy (illustrative; scope down to your bucket + prefix):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<bucket>",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "terraform-ncp3/sandbox-eks/*"
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::<bucket>/terraform-ncp3/sandbox-eks/terraform.tfstate"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<bucket>/terraform-ncp3/sandbox-eks/terraform.tfstate.tflock"
    }
  ]
}

---

3) Initialize the backend (and migrate state if it exists)

Directory: terraform/

3.1 Create a backend config file (recommended for Option A)

Create terraform/backend.hcl (do not commit):

bucket = "<bucket>"
key    = "terraform-ncp3/sandbox-eks/terraform.tfstate"

# Optional: if you use AWS shared config profiles
profile = "<profile>" # e.g. terraform

Then initialize (from within terraform/):

terraform init -reconfigure -backend-config=backend.hcl

If you rely on TF_VAR_aws_profile for the provider, remember the backend won’t read that; set AWS_PROFILE too (or set profile = ... above).

After the first successful init:

• Verify terraform/.terraform.lock.hcl exists and commit it (it pins provider selections for reproducible runs).

3.2 If an existing local state file exists, migrate it instead of importing

If someone already has the correct local state file from previous applies, the safest path is to migrate it:

terraform init -reconfigure -migrate-state -backend-config=backend.hcl

This preserves all resource addresses and avoids manual terraform import for most cases.

---

4) Final step: discover existing resources and import into the new remote state

This section applies if:

• The old local .tfstate is missing / not trustworthy, or
• You want to double-check that the remote state fully captures what already exists.

4.1 Guardrails before importing

• Do not run terraform apply until imports are complete and terraform plan looks sane.
• Ensure the Terraform variables match what was used to create the infra:
  • aws_region should be eu-central-1
  • cluster_name default is sandbox-eks
  • Node group name is ${cluster_name}-nodes (default sandbox-eks-nodes)
• Subnet resources use count and depend on the first two AZ names returned by AWS. Import indices must match the AZ suffix in the subnet Name tag.

4.2 Discover IDs in AWS (examples)

Use tags from the config (notably Name = "${var.cluster_name}-...") to locate resource IDs:

🔎 Example AWS CLI discovery commands (eu-central-1)

REGION=eu-central-1
CLUSTER=sandbox-eks

# VPC
aws ec2 describe-vpcs --region "$REGION" \
  --filters "Name=tag:Name,Values=${CLUSTER}-vpc" \
  --query 'Vpcs[0].VpcId' --output text

# Internet Gateway
aws ec2 describe-internet-gateways --region "$REGION" \
  --filters "Name=tag:Name,Values=${CLUSTER}-igw" \
  --query 'InternetGateways[0].InternetGatewayId' --output text

# Subnets
aws ec2 describe-subnets --region "$REGION" \
  --filters "Name=tag:Name,Values=${CLUSTER}-public-*" \
  --query 'Subnets[].{id:SubnetId,az:AvailabilityZone,name:Tags[?Key==`Name`]|[0].Value}' --output table

aws ec2 describe-subnets --region "$REGION" \
  --filters "Name=tag:Name,Values=${CLUSTER}-private-*" \
  --query 'Subnets[].{id:SubnetId,az:AvailabilityZone,name:Tags[?Key==`Name`]|[0].Value}' --output table

# EIP allocation id (tagged)
aws ec2 describe-addresses --region "$REGION" \
  --filters "Name=tag:Name,Values=${CLUSTER}-nat-eip" \
  --query 'Addresses[0].AllocationId' --output text

# NAT gateway
aws ec2 describe-nat-gateways --region "$REGION" \
  --filter "Name=tag:Name,Values=${CLUSTER}-nat" \
  --query 'NatGateways[0].NatGatewayId' --output text

# Route tables
aws ec2 describe-route-tables --region "$REGION" \
  --filters "Name=tag:Name,Values=${CLUSTER}-public-rt" \
  --query 'RouteTables[0].RouteTableId' --output text

aws ec2 describe-route-tables --region "$REGION" \
  --filters "Name=tag:Name,Values=${CLUSTER}-private-rt" \
  --query 'RouteTables[0].RouteTableId' --output text

# EKS
aws eks describe-cluster --region "$REGION" --name "$CLUSTER" \
  --query 'cluster.name' --output text

aws eks list-nodegroups --region "$REGION" --cluster-name "$CLUSTER"
aws eks list-addons --region "$REGION" --cluster-name "$CLUSTER"

# IAM roles
aws iam get-role --role-name "${CLUSTER}-cluster-role" --query 'Role.RoleName' --output text
aws iam get-role --role-name "${CLUSTER}-node-role" --query 'Role.RoleName' --output text

4.3 Import commands (addresses + ID formats)

Import order is mostly for human sanity; Terraform doesn’t always require strict ordering, but this tends to minimize confusing errors.

📥 Import checklist for this repo’s current resources

Replace IDs with the discovered values.

Networking (terraform/vpc.tf)

terraform import aws_vpc.this vpc-xxxxxxxx
terraform import aws_internet_gateway.this igw-xxxxxxxx

# Public subnets (map [0]/[1] to the AZ suffix in the Name tag)
terraform import 'aws_subnet.public[0]' subnet-aaaaaaaa
terraform import 'aws_subnet.public[1]' subnet-bbbbbbbb

# Private subnets
terraform import 'aws_subnet.private[0]' subnet-cccccccc
terraform import 'aws_subnet.private[1]' subnet-dddddddd

terraform import aws_eip.nat eipalloc-xxxxxxxx
terraform import aws_nat_gateway.this nat-xxxxxxxx

terraform import aws_route_table.public rtb-xxxxxxxx
terraform import aws_route_table.private rtb-yyyyyyyy

# Route table associations use: <subnet-id>/<route-table-id>
terraform import 'aws_route_table_association.public[0]'  subnet-aaaaaaaa/rtb-xxxxxxxx
terraform import 'aws_route_table_association.public[1]'  subnet-bbbbbbbb/rtb-xxxxxxxx
terraform import 'aws_route_table_association.private[0]' subnet-cccccccc/rtb-yyyyyyyy
terraform import 'aws_route_table_association.private[1]' subnet-dddddddd/rtb-yyyyyyyy

IAM (terraform/iam.tf)

terraform import aws_iam_role.eks_cluster sandbox-eks-cluster-role
terraform import aws_iam_role.node_group sandbox-eks-node-role

terraform import aws_iam_role_policy_attachment.eks_cluster_policy \
  'sandbox-eks-cluster-role/arn:aws:iam::aws:policy/AmazonEKSClusterPolicy'

terraform import aws_iam_role_policy_attachment.node_group_worker_policy \
  'sandbox-eks-node-role/arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy'

terraform import aws_iam_role_policy_attachment.node_group_cni_policy \
  'sandbox-eks-node-role/arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy'

terraform import aws_iam_role_policy_attachment.node_group_ecr_readonly \
  'sandbox-eks-node-role/arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'

EKS (terraform/eks.tf)

terraform import aws_eks_cluster.this sandbox-eks

# Node group import id format: <cluster-name>:<node-group-name>
terraform import aws_eks_node_group.this sandbox-eks:sandbox-eks-nodes

# Addons import id format: <cluster-name>:<addon-name>
terraform import aws_eks_addon.coredns    sandbox-eks:coredns
terraform import aws_eks_addon.kube_proxy sandbox-eks:kube-proxy
terraform import aws_eks_addon.vpc_cni    sandbox-eks:vpc-cni

4.4 Verify state matches reality

After imports:

• terraform plan should ideally be a no-op.
• If there are diffs, decide whether:
  • The config should change to match reality, or
  • Reality should be changed via terraform apply.

Common import-related issues to watch for:

• Subnet indices swapped (AZ order mismatch) → Terraform will want to recreate subnets.
• EKS add-on settings drifted (Terraform will propose updates).

---

5) Validation / acceptance criteria

• terraform init succeeds with the S3 backend and use_lockfile = true.
• State file exists in S3 at the expected key, and a .tflock object appears briefly during a plan/apply.
• .gitignore prevents state and .terraform/ from being committed (including terraform/backend.hcl).
• terraform/.terraform.lock.hcl is committed.
• nix develop provides terraform and its version is >= 1.11.
• After import/migration, terraform plan does not propose recreating the existing cluster/VPC (only expected drift).

---

Generated with mux • Model: anthropic:claude-opus-4-6 • Thinking: xhigh • Cost: $4.83

[ThomasK33]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eeb2249fea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed feedback: subnet newbits are now derived dynamically from var.vpc_cidr prefix length (24 - prefix), with a validation that the VPC CIDR must be shorter than /24. This ensures correct /24 subnets for any VPC size.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 431232c6a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Tightened VPC CIDR validation to <= /19 so that all subnet network numbers (up to 20) are guaranteed to fit within 2^(24-prefix). A /20 VPC (which only has 4 new bits → max netnum 15) is now correctly rejected.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 👍

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

https://mux.md/VWLYq#VELsHjlNqgtJ5A