Add EKS Auto Mode deployment mode

[Erik Weathers (erikdw)]

Add EKS Auto Mode deployment mode

Summary

Adds a create_eks_cluster = true deployment mode that provisions a complete Braintrust dataplane on EKS Auto Mode instead of the existing Lambda + EC2 path. All Braintrust workloads (API, brainstore reader / fastreader / writer) run in-cluster as pods and are deployed via the Braintrust Helm chart.

When enabled, the module owns:

• A new EKS Auto Mode cluster (${deployment_name}-eks)
• All cluster + node IAM roles
• Pod Identity associations binding Kubernetes service accounts to AWS IAM roles
• A custom Karpenter NodePool/NodeClass constraining Brainstore to NVMe-backed instance families (delivered via a small in-repo Helm chart)
• A pre-created NLB and CloudFront VPC Origin, adopted by the AWS Load Balancer Controller at Helm release time
• The Braintrust Helm release itself, with values rendered from Terraform

Everything else (VPC, RDS, ElastiCache, S3, KMS, API/Brainstore IAM) is shared with the existing Lambda/EC2 path.

The Lambda, EC2 Brainstore, and Lambda-URL ingress submodules are disabled in this mode (gated by use_deployment_mode_external_eks = true, which create_eks_cluster = true requires).

Why EKS Auto Mode

Auto Mode lets AWS manage the control plane add-ons (VPC CNI, CoreDNS, kube-proxy, Pod Identity Agent, AWS Load Balancer Controller, EBS CSI driver) and node lifecycle (via a managed Karpenter). The same capabilities on self-managed EKS mean owning the install, IAM configuration, version-compatibility matrix, and upgrade choreography for each addon — plus whichever subset of {Karpenter, Pod Identity Agent, EBS CSI, metrics-server} matches your feature choices. None of it is individually hard; collectively it's real recurring work on every cluster upgrade. Auto Mode hands that coordination surface to AWS in exchange for the managed-mode premium and some lost flexibility. This module therefore uses Auto Mode exclusively rather than self-managed EKS or the terraform-aws-modules/eks community module.

Usage

Minimal example

See examples/braintrust-data-plane-eks/ for the production-sized canonical config, or examples/braintrust-data-plane-eks-sandbox/ for a cheap disposable sandbox variant (smaller RDS, Redis, and a values.yaml alongside that shrinks the chart components to 1-replica with tight CPU/memory). The shortest working invocation:

module "braintrust-data-plane" {
  source = "../../"

  deployment_name        = "my-deployment"
  braintrust_org_name    = "my-org"
  brainstore_license_key = var.brainstore_license_key

  use_deployment_mode_external_eks = true
  create_eks_cluster               = true

  helm_chart_version = "6.1.0"

  # ...plus the usual postgres_*, redis_*, etc.
}

Single-apply bootstrap

terraform apply. One command.

Cold first-deploy runtime is ~15 minutes (cluster ~8-10, then RDS + Redis + Helm release). Subsequent applies are incremental.

Two design choices make the one-command path work:

1. Provider config is sourced from module outputs, not data.aws_eks_cluster. The example's provider.tf reads module.braintrust-data-plane.eks_cluster_endpoint, eks_cluster_ca_certificate_data, and eks_cluster_name directly off the module. Terraform treats these as "known after apply" on the first run and defers provider resolution until the cluster exists. A data source, by contrast, reads at refresh (pre-plan) and would fail on a fresh deploy — that was the reason the first iteration of this module required a -target'd two-step apply.
2. NodeClass + NodePool are delivered via helm_release, not kubernetes_manifest. kubernetes_manifest reads CRD schemas from the live cluster at plan time to validate manifests, which fails on a fresh deploy; Helm renders templates locally and applies at apply time, with no plan-time cluster dependency. The CRDs live in an in-repo chart at modules/eks-deploy/charts/brainstore-nodepool/.

Architecture

Module layout

New submodules:

• modules/eks-cluster/ — EKS cluster, cluster+node IAM roles, NLB pre-creation, CloudFront VPC Origin wiring, CloudFront distribution.
• modules/eks-deploy/ — the Kubernetes / Helm layer: namespace, braintrust-secrets Secret, Pod Identity associations, the brainstore-nodepool helm release (NodeClass + NodePool), and the braintrust helm release itself.

New in-repo Helm chart:

• modules/eks-deploy/charts/brainstore-nodepool/ — tiny chart with just two templates (NodeClass + NodePool). Not published anywhere; lives with the Terraform source so the module is self-contained.

Top-level eks.tf wires the three submodules together. Root-level main.tf is touched only lightly (for services_common to receive the EKS cluster ARN for Pod Identity trust scoping).

Module ordering

module.eks_cluster → module.services_common → module.eks_deploy

1. eks_cluster provisions the cluster and exports its ARN.
2. services_common builds the API + Brainstore IAM roles with Pod Identity trust policies scoped to (cluster_arn, namespace, service_account).
3. eks_deploy creates the Pod Identity associations binding SAs to roles, plus the namespace / Secret / brainstore-nodepool chart / Braintrust helm release.

This is why the EKS layer is split into two submodules rather than one: services_common is also used by the non-EKS path, so it can't live inside eks_deploy, and the role ARNs it produces are consumed by eks_deploy, so services_common can't live inside eks_cluster.

Key design decisions

• Pod Identity, not IRSA. Auto Mode ships the Pod Identity Agent preinstalled. Pod Identity uses simpler trust policies, supports session tags, and doesn't require an OIDC provider. The module creates aws_eks_pod_identity_association resources for both the braintrust-api and brainstore service accounts. The chart still writes an IRSA-style eks.amazonaws.com/role-arn annotation on the service accounts; this is harmless because Pod Identity intercepts AWS SDK credential resolution before IRSA is consulted.

• Pre-created NLB adopted by the Load Balancer Controller. The CloudFront VPC Origin needs the NLB ARN at plan time, but the Load Balancer Controller normally creates NLBs on demand when a Service becomes type: LoadBalancer. The module pre-creates the NLB in Terraform (aws_lb.api), and the chart's Service uses service.beta.kubernetes.io/aws-load-balancer-name + aws-load-balancer-security-groups annotations to have the controller adopt the existing NLB rather than create a new one. Security groups can only be attached to an NLB at creation time, which is why the NLB SG is also owned by Terraform.

• Custom Brainstore NodePool. Brainstore caches to local NVMe SSD via emptyDir, so its pods need NVMe-backed EC2 families (c8gd, c7gd, m7gd, etc.). Auto Mode's default general-purpose NodePool doesn't constrain to those families, so the module adds a custom NodeClass + NodePool that does, and Brainstore pods target it via the braintrust.dev/node-pool: brainstore nodeSelector injected into the Helm values.

• NodePool delivered via helm_release, not kubernetes_manifest. kubernetes_manifest reads CRD schemas from the live cluster at plan time. That's incompatible with single-apply bootstrap because the cluster doesn't exist yet on the first plan. Wrapping the two manifests in a tiny local Helm chart moves the cluster contact to apply time. The rendered objects are structurally identical to what kubernetes_manifest produced — verified by rendering the chart and diffing field-by-field against the old values, including the tricky aws:eks:cluster-name colon-key.

• Provider config from module outputs, not data.aws_eks_cluster. The example's provider.tf reads eks_cluster_endpoint, eks_cluster_ca_certificate_data, and eks_cluster_name directly off the module. Terraform treats module outputs that trace back to "known after apply" resource attributes as unknown at plan time and defers provider resolution until the cluster exists. A data source reads at refresh (pre-plan) and would fail on a fresh deploy.

• Exec auth for the Kubernetes/Helm providers, not static tokens. The example's provider.tf uses exec { aws eks get-token } rather than the simpler aws_eks_cluster_auth data source. The static-token pattern expires after 15 minutes — short enough to fail if an apply sits at an approval prompt or if the operator walks away between terraform plan and terraform apply. Exec auth refreshes on every API call and requires only the AWS CLI on the runner (which consumers need anyway, for aws eks update-kubeconfig).

Destroy choreography

TBD whether to ship this in the PR. The prepare_for_destroy mechanism below is implemented and validated, but it overlaps with the chart-level annotation already shipped in fc11624. If the chart annotation reliably propagates to the live TG on every supported chart version, this is belt-and-suspenders and could be cut to keep the surface area smaller. Keeping it earns its place when the chart annotation drifts (older chart, manual overrides, broken-state cluster) — exactly the case that motivated the manual kubectl patch runbook in TROUBLESHOOTING.md and produced an orphan TG on a real sandbox teardown. Decide before merge.

Tearing down an EKS-mode deployment is a two-step apply→destroy:

1. Set prepare_for_destroy = true and run terraform apply.
2. Run terraform destroy.

The preflight resources live in modules/eks-deploy/main.tf behind count = var.prepare_for_destroy ? 1 : 0:

• kubernetes_annotations.api_drain_zero patches service.beta.kubernetes.io/aws-load-balancer-target-group-attributes on the api Service to deregistration_delay.timeout_seconds=0. Belt-and-suspenders: the chart's helm-values.yaml.tpl already sets this, but this resource forces the live annotation back to the right value if it ever drifted (older chart, manual override, broken state).
• terraform_data.api_tg_drain_zero calls aws elbv2 modify-target-group-attributes directly on every TargetGroup tagged BraintrustDeploymentName=<deployment_name>. Faster path than waiting for the LB Controller's reconcile loop to propagate the annotation, and works even on TGs created before the annotation was set.

With drain wait at zero, the LB Controller releases its service.eks.amazonaws.com/resources finalizer the moment helm uninstall deletes the api Service, finishes its own TG cleanup, and helm_release.braintrust returns in seconds. No kubectl patch workarounds, no orphan TargetGroups left in AWS.

Why this exists: in earlier sandbox tear-downs, terraform destroy froze for ~5 min on the helm_release while LBC waited out the default 300s drain timer. The manual workaround (kubectl -n braintrust patch svc braintrust-api --type merge -p '{"metadata":{"finalizers":null}}') unblocks the destroy but interrupts the controller mid-cleanup, leaving an orphan TG behind. prepare_for_destroy is the supported alternative.

Scope: service infra only. Data-bearing resources have separate, explicit knobs:

• RDS — DANGER_disable_database_deletion_protection = true (existing) flips the RDS deletion_protection attribute. Required for terraform destroy to remove the database; intentionally not bundled into prepare_for_destroy.
• S3 — buckets are deliberately not exposed as destroyable from Terraform. No force_destroy toggle, no DANGER_* flag. If you need to tear a sandbox down completely, empty the buckets manually first (aws s3 rm s3://<bucket> --recursive, then aws s3api delete-objects for non-current versions and delete-markers if versioning was enabled). The cost of a stray destroy hitting a data bucket is too high to mitigate with a flag.

New variables

All defaulted except helm_chart_version when create_eks_cluster = true.

Variable	Default	Purpose
create_eks_cluster	false	Master switch for this mode. Requires use_deployment_mode_external_eks = true.
eks_kubernetes_version	"1.31"	Kubernetes version for the EKS cluster.
eks_brainstore_nodepool_instance_families	["c8gd", "c7gd", "m7gd"]	EC2 families Karpenter may pick from for Brainstore nodes. Must be NVMe-backed.
helm_chart_version	null	Required when create_eks_cluster = true. No default so chart upgrades are always deliberate.
eks_helm_values_file	null	Path to a YAML file with Helm values overrides, merged in after the module's rendered defaults. Idiomatic usage: eks_helm_values_file = "${path.module}/values.yaml" so the file lives alongside your main.tf. Leave null to accept chart defaults. See the braintrust-data-plane-eks-sandbox example for sandbox-sized values.
prepare_for_destroy	false	Pre-flight before terraform destroy. Flip true, apply, then destroy. Zeroes deregistration_delay on the LB Controller's TargetGroup(s) so the finalizer doesn't hang helm_release.braintrust on destroy and the controller cleans up its own TGs (no orphans). EKS-mode only. See Destroy choreography below and TROUBLESHOOTING.md.

New module outputs

The three starred outputs below are required by the example's provider.tf to configure the kubernetes/helm providers from module outputs instead of a data.aws_eks_cluster lookup — which is what enables single-apply bootstrap. The rest are broadly useful for downstream consumers wiring this module into larger deployments (IAM references for external Pod Identity associations, Postgres/Redis connection details for downstream Kubernetes Secret construction, S3 bucket names for downstream IAM policy templates, NLB identifiers, etc.).

Output	Sensitive	Purpose
★ eks_cluster_name	no	Cluster name (used in the aws eks get-token exec arg in provider.tf).
★ eks_cluster_endpoint	no	API server endpoint for the kubernetes/helm providers' host.
★ eks_cluster_ca_certificate_data	yes	Base64-encoded cluster CA. Consumed by the kubernetes/helm providers' cluster_ca_certificate (after base64decode()).
eks_cluster_security_group_id	no	Cluster SG (attached to Auto Mode nodes). Useful for authoring additional inbound rules from external sources.
eks_nlb_arn	no	ARN of the pre-created internal NLB adopted by the LB Controller.
eks_nlb_name	no	NLB name (referenced by the chart's aws-load-balancer-name annotation).
nlb_security_group_id	no	Security group attached to the NLB.
code_bundle_bucket_id	no	S3 bucket for code bundles.
lambda_responses_bucket_id	no	S3 bucket for lambda responses.
postgres_database_address	no	Postgres hostname.
postgres_database_port	no	Postgres port.
redis_endpoint	no	Redis hostname.
redis_port	no	Redis port.
api_handler_role_arn	no	IAM role ARN for the braintrust-api service account.
brainstore_iam_role_arn	no	IAM role ARN for the brainstore service account (also the EC2 role on the EC2-Brainstore path).

★ = required for the single-apply provider.tf pattern.

Module ↔ Helm chart contract

The module and chart are tightly coupled — several names, ports, keys, and paths have to match exactly on both sides. The full list is documented in CONTRACT.md (tested chart version 6.1.0, supported range 6.x). Highlights:

• K8s Secret name braintrust-secrets and its keys (PG_URL, REDIS_URL, FUNCTION_SECRET_KEY, BRAINSTORE_LICENSE_KEY)
• Service account names (braintrust-api, brainstore) used in both the Pod Identity associations and the chart
• API container port 8000 — used by the CloudFront VPC Origin and by the cluster SG ingress rule that admits NLB traffic
• The four aws-load-balancer-* service annotations the controller reads to adopt our pre-created NLB, plus aws-load-balancer-additional-resource-tags for deployment-scoped tagging of controller-created resources
• Brainstore nodeSelector label braintrust.dev/node-pool: brainstore

Any of these moving or renaming on the chart side breaks us, often silently. Drift detection between the module and chart is a follow-up (below).

Tradeoffs accepted with single-apply bootstrap

Single-apply is strictly an improvement if the target audience can handle the failure modes below. For Braintrust's self-hosted data plane audience (sophisticated operators), the judgment is that it is. For less-experienced consumers, two-step would have been safer. This PR accepts the tradeoff.

#	Tradeoff	Likelihood	Severity	Recovery
1	Out-of-band cluster deletion (AWS console, cleanup script, aws eks delete-cluster) breaks terraform plan at refresh — the provider can no longer read eks_cluster_endpoint.	Medium	Medium	terraform state rm the kubernetes_* and helm_release.* resources, then terraform apply to recreate. Full runbook in RECOVERY.md.
2	-targeted partial destroy of just the cluster orphans K8s state with no way to reach it.	Low (anti-pattern)	Medium	Same as #1.
3	"Known after apply" values in provider config generate warnings on first plan. Terraform ≥1.3 tolerates this, but future provider/TF versions could tighten the rules.	Low (fine today, future risk)	Low	Revert the provider config to a data.aws_eks_cluster lookup + -target two-step. Change is reversible.
4	Helm-managed NodeClass/NodePool. If the helm release secret for brainstore-nodepool gets corrupted or a customer kubectl deletes the CRs out of band, recovery is rougher than before.	Low	Low	helm uninstall brainstore-nodepool -n braintrust + terraform apply.
5	Local chart templates don't validate at plan time. Syntax errors in charts/brainstore-nodepool/templates/*.yaml fail at apply time, not plan.	Low	Low	Fix chart, re-apply. Could add helm lint to CI later.
6	CRD availability race — helm_release.brainstore_nodepool could theoretically try to create a NodeClass / NodePool before Auto Mode finishes installing the Karpenter CRDs.	Very low	Low	Retry apply. Not observed; Auto Mode CRDs are present by the time the cluster is kubectl-reachable.

In-band terraform destroy still works correctly via the dependency graph — Terraform drains K8s resources first, then the cluster.

Known limitations / follow-ups

• TG naming. Controller-generated TG name k8s-braintru-braintru-* still collides visually across deployments. The additional-resource-tags fix makes the tags disambiguating, but the name itself isn't configurable on the controller side. Low priority.
• Drift detection between module and chart. CONTRACT.md enumerates the coupling surfaces, but there's no automated check. A CI smoke test that renders the chart against the module's template values and grep-asserts the known-good keys would prevent silent breakage on chart upgrades. Deferred.
• Dedicated node per Brainstore pod. The EC2-Brainstore path runs each Brainstore component on its own instance; on EKS today multiple Brainstore pods can land on the same Karpenter-provisioned node (as we observed in testing — all 3 readers + the api pod on one c8gd.*). That's fine for sandbox throughput but defeats the isolation/headroom story of the EC2 path for production workloads. Fix lives in the Helm chart (pod anti-affinity on braintrust.dev/brainstore-role), not in this module. Deferred.
• Explicit NodePool for the API component. Today the API pod has no nodeSelector and falls through to Auto Mode's default general-purpose NodePool, with opaque instance-family selection managed by AWS. Brainstore, by contrast, targets the module-owned brainstore NodePool pinned to NVMe-Graviton families. Parallel follow-up: add a second custom NodePool (api) in the local chart with its own eks_api_nodepool_instance_families variable (default non-NVMe Graviton compute: c8g/c7g/m7g). Brings the API pod under the same explicit control as Brainstore — predictable instance selection, consistent operational model, and enables per-pool tuning of disruption/consolidation policy (API pods tolerate more aggressive consolidation than Brainstore). Implementation is ~50 lines of chart YAML + one variable + one helm-values-template edit; guard the nodeSelector rendering on the pool being enabled to avoid a stuck-Pending footgun. Deferred.

Testing

End-to-end validated in a sandbox AWS account:

• Fresh single-terraform apply succeeds from an empty AWS account.
• All 4 pods (braintrust-api, brainstore-fastreader, brainstore-reader, brainstore-writer) reach Running 1/1.
• Brainstore pods land on a Karpenter-provisioned node in the custom NodePool with an NVMe-backed instance type (c8gd.xlarge observed).
• curl https://<cloudfront-domain>/ returns 200 OK + Hello World! from the API through CloudFront → NLB → pod.
• Subsequent applies are idempotent and don't recreate the NLB or CloudFront.
• Verified TG carries the BraintrustDeploymentName tag after the tagging commit.
• Verified coexistence with a separate manual dataplane install in the same AWS account — no cross-deployment interference at the AWS resource layer (tags and cluster-scoping keep them isolated).
• Verified the brainstore-nodepool chart renders to the exact same Kubernetes manifests as the previous kubernetes_manifest resources (field-by-field JSON diff, including the aws:eks:cluster-name colon-key YAML parse).
• Ran smoke test from a real Braintrust org in the braintrust.dev UI:
  • Configured to use the CloudFront API URL
  • Created a service token
  • Successfully interacted with ChatGPT from the Playground