diff --git a/AGENTS.md b/AGENTS.md index 07c296d2..f6d7dcc9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,9 @@ This is a Terraform module that deploys the Braintrust hybrid data plane on AWS. │ ├── braintrust-data-plane/ # Production example │ ├── braintrust-data-plane-sandbox/ # Sandbox/testing example │ ├── braintrust-data-plane-external-eks-quarantine/ -│ └── cloudfront-logging/ +│ ├── cloudfront-logging/ +│ ├── eks-auto-mode/ # EKS Auto Mode example with Terraform-managed Helm deployment +│ └── eks-terraform-managed/ # EKS example with Terraform-managed node groups and manual Helm install ├── scripts/ # Operational helper scripts └── mise.toml # Tool versions and tasks (terraform, tflint, uv) ``` diff --git a/README.md b/README.md index 8504c11e..30174bd9 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,54 @@ If you're using a brand new AWS account for your Braintrust data plane you will ## Module Configuration All module input variables and outputs are documented inline in the module's Terraform files (see `variables.tf`, `outputs.tf`, and the submodules for details). +## EKS Deployment Mode + +Use `use_deployment_mode_eks = true` together with `create_eks_cluster = true` to have this module build the AWS and EKS infrastructure for a Braintrust deployment. In this mode, the root module stops at the infrastructure boundary: it creates the VPC-facing resources, database, Redis, storage, IAM, EKS cluster, node capacity, and the outputs needed by a Kubernetes deployment layer. The bundled CloudFront + private NLB ingress path is enabled by default and can be disabled with `eks_enable_cloudfront_nlb_ingress = false` if you want to bring your own ingress. + +The two EKS examples show the intended compositions on top of that base: + +- [`examples/eks-auto-mode`](examples/eks-auto-mode) is the one-click path. It composes the root module with `modules/eks-deploy` to create the cluster, bundled CloudFront/NLB ingress, and deploy the Braintrust app in the same apply, using EKS Auto Mode for compute. +- [`examples/eks-terraform-managed`](examples/eks-terraform-managed) is the handoff path. It creates the AWS data plane and a Terraform-managed EKS cluster with explicit node groups, disables the bundled ingress by default, and leaves the final Braintrust chart install plus ingress choice to the customer. + +The older `use_deployment_mode_external_eks` variable remains supported as a deprecated alias for backwards compatibility. + +The module enables the private EKS API endpoint by default. If you keep the +public endpoint enabled, set `eks_public_access_cidrs` to explicit operator or +CI egress CIDRs for production instead of leaving it open to `0.0.0.0/0`. The +open default is allowed for compatibility and bootstrap convenience. If you +disable public access, run Terraform, `kubectl`, and Helm steps that contact the +cluster from the VPC or a connected network. + +Use `eks_access_entries` to grant long-lived human or CI access to module-managed +clusters. The cluster creator receives bootstrap admin access, but additional +operators should be represented explicitly with EKS access entries and scoped +access policies. + +If your EKS cluster is managed outside this module and you need a module-managed +quarantine VPC for user-defined functions, use the external EKS quarantine +example: + +- [`examples/braintrust-data-plane-external-eks-quarantine`](examples/braintrust-data-plane-external-eks-quarantine) for EKS with a quarantine VPC for user-defined functions + +By default, EKS deployments with `eks_enable_cloudfront_nlb_ingress = true` use the generated CloudFront `*.cloudfront.net` hostname and the default CloudFront TLS certificate. Leave `custom_domain = null` and `custom_certificate_arn = null` for this baseline. + +To use a customer-owned hostname with the bundled ingress path, set `custom_domain` and provide an ACM certificate ARN from `us-east-1` through `custom_certificate_arn`. CloudFront requires viewer certificates for custom aliases to live in `us-east-1`, even when the data plane runs in another AWS region. + +The module does not create or validate the certificate for you, and it does not create DNS records automatically. The expected workflow is: + +1. Request or import an ACM certificate for the exact hostname you want CloudFront to serve, in `us-east-1`. +2. Validate that certificate in ACM. +3. Set `custom_domain` and `custom_certificate_arn` on the module. +4. Apply Terraform so the CloudFront distribution is updated with the alias and viewer certificate. +5. Create a DNS alias record for `custom_domain` that points at the module outputs `cloudfront_distribution_domain_name` and `cloudfront_distribution_hosted_zone_id`. +6. Register the resulting `api_url` output in the Braintrust dashboard. + +If you leave both values null, the module continues to use the generated CloudFront hostname and default certificate. If you set `custom_domain`, you should also set `custom_certificate_arn`; CloudFront will reject a custom alias without a matching `us-east-1` ACM certificate. + +If you set `eks_enable_cloudfront_nlb_ingress = false`, the module does not create the private NLB, CloudFront distribution, or related CloudFront outputs. In that mode, `api_url` is also null until you provide your own ingress path. + +`waf_acl_id` is optional in both the Lambda/API Gateway and EKS/CloudFront paths. If set, the module attaches your existing WAF Web ACL to the CloudFront distribution; if not set, no WAF is attached. + ## Useful scripts ### dump-logs.sh diff --git a/examples/README.md b/examples/README.md index 5f5d9057..e8ef617b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -7,6 +7,8 @@ Example configurations for deploying the Braintrust data plane on AWS. Copy the | [braintrust-data-plane](./braintrust-data-plane/) | Standard production-sized deployment. Start here for most use cases. | | [braintrust-data-plane-sandbox](./braintrust-data-plane-sandbox/) | Minimal deployment for infrastructure testing. Downsized instances, no quarantine VPC, deletion protection disabled. Not for workload testing. | | [braintrust-data-plane-external-eks-quarantine](./braintrust-data-plane-external-eks-quarantine/) | Deployment with an external EKS cluster and quarantine VPC for user-defined functions. | +| [eks-terraform-managed](./eks-terraform-managed/) | Terraform-managed node-group cluster example. Terraform creates the AWS data plane and EKS cluster, disables the bundled CloudFront/NLB ingress by default, and leaves the app install and ingress choice to the customer. | +| [eks-auto-mode](./eks-auto-mode/) | Full one-click Auto Mode example. Terraform creates the AWS data plane, EKS cluster, bundled CloudFront/NLB ingress, Kubernetes bootstrap resources, and the Braintrust application deployment. | | [cloudfront-logging](./cloudfront-logging/) | Add-on example for enabling CloudFront access logging to S3. | See the [self-hosting documentation](https://www.braintrust.dev/docs/guides/self-hosting/aws) for full setup instructions. diff --git a/examples/braintrust-data-plane-external-eks-quarantine/README.md b/examples/braintrust-data-plane-external-eks-quarantine/README.md index 903e5405..9ea5d05b 100644 --- a/examples/braintrust-data-plane-external-eks-quarantine/README.md +++ b/examples/braintrust-data-plane-external-eks-quarantine/README.md @@ -2,13 +2,13 @@ This is an example configuration for deploying a Braintrust data plane with the following characteristics: -- **External EKS Deployment Mode**: Uses `use_deployment_mode_external_eks = true` to deploy services on an existing EKS cluster managed outside of Terraform +- **External EKS Deployment Mode**: Uses `use_deployment_mode_eks = true` to deploy services on an existing EKS cluster managed outside of Terraform - **EKS Pod Identity**: Enables `enable_eks_pod_identity = true` for the Braintrust IAM roles - **Quarantine VPC**: Deploys the Quarantine VPC (`enable_quarantine_vpc = true`) for running user-defined functions in an isolated environment ## Quarantine VPC IAM Permissions -The Quarantine VPC IAM permissions are automatically deployed when `enable_quarantine_vpc = true`, even when using `use_deployment_mode_external_eks = true`. These IAM resources are created in the `services-common` module: +The Quarantine VPC IAM permissions are automatically deployed when `enable_quarantine_vpc = true`, even when using `use_deployment_mode_eks = true`. These IAM resources are created in the `services-common` module: - `QuarantineInvokeRole` - Role used by the API handler to invoke quarantined functions - `QuarantineFunctionRole` - Role used by the quarantined Lambda functions @@ -21,7 +21,7 @@ These resources are available for use by your EKS-deployed services via EKS Pod ### Required Settings 1. **EKS Configuration**: - - Set `use_deployment_mode_external_eks = true` + - Set `use_deployment_mode_eks = true` - Set `enable_eks_pod_identity = true` 2. **Quarantine VPC**: @@ -123,3 +123,4 @@ Ensure these don't conflict with: - See the main [README.md](../../README.md) for general information - See [examples/braintrust-data-plane/README.md](../braintrust-data-plane/README.md) for standard deployment examples +- See [helm/braintrust/examples/aws-eks-quarantine/values.yaml](../../../helm/braintrust/examples/aws-eks-quarantine/values.yaml) for a matching Helm values example diff --git a/examples/braintrust-data-plane-external-eks-quarantine/main.tf b/examples/braintrust-data-plane-external-eks-quarantine/main.tf index 3dcc88c9..c9f9190e 100644 --- a/examples/braintrust-data-plane-external-eks-quarantine/main.tf +++ b/examples/braintrust-data-plane-external-eks-quarantine/main.tf @@ -4,15 +4,13 @@ # with Quarantine VPC enabled and EKS Pod Identity. # # This configuration: -# - Uses external EKS deployment mode (use_deployment_mode_external_eks = true) +# - Uses external EKS deployment mode (use_deployment_mode_eks = true) # - Enables EKS Pod Identity (enable_eks_pod_identity = true) # - Deploys the Quarantine VPC (enable_quarantine_vpc = true) # - Deploys all IAM permissions needed for the Quarantine VPC module "braintrust-data-plane" { - # Using local source for testing - change to GitHub source for production - source = "../../" - # source = "github.com/braintrustdata/terraform-braintrust-data-plane" + source = "github.com/braintrustdata/terraform-braintrust-data-plane" # Append '?ref=' to lock to a specific version of the module. # This is primarily used for tagging and naming resources in your AWS account. @@ -28,7 +26,7 @@ module "braintrust-data-plane" { # Enable external EKS deployment mode # When true, disables lambdas, ec2, and ingress submodules. # It assumes an EKS deployment is being done outside of terraform. - use_deployment_mode_external_eks = true + use_deployment_mode_eks = true # With external EKS, there are additional configurations that must be applied after the EKS cluster has been created outside of this module. # Enable EKS Pod Identity for the Braintrust IAM roles diff --git a/examples/eks-auto-mode/README.md b/examples/eks-auto-mode/README.md new file mode 100644 index 00000000..d5949b8f --- /dev/null +++ b/examples/eks-auto-mode/README.md @@ -0,0 +1,161 @@ +# Braintrust Data Plane On EKS Auto Mode + +This is the full one-click EKS Auto Mode example. + +Terraform creates: + +- the Braintrust AWS data plane resources +- the EKS cluster +- the CloudFront edge and private NLB +- the Kubernetes bootstrap resources +- the AWS Load Balancer Controller +- the Brainstore Auto Mode NodeClass and NodePool resources +- the Braintrust Helm release + +If you want Terraform-managed node groups but a manual app install, use +[`../eks-terraform-managed/`](../eks-terraform-managed/). + +## Required Inputs + +- `braintrust_org_name` +- `brainstore_license_key` +- `eks_helm_chart_version` + +## EKS API Endpoint Access + +This example manages Kubernetes and Helm resources during the same Terraform +apply that creates the EKS cluster. If you disable the public EKS API endpoint, +the Terraform runner must be inside the VPC or on a connected network that can +reach the private endpoint, such as VPN, Direct Connect, Transit Gateway, a +VPC-attached CI runner, or an SSM-reached admin host. + +For a laptop or public CI bootstrap, keep private access enabled. The example +defaults `eks_public_access_cidrs` to `["0.0.0.0/0"]` for compatibility, but +production deployments should restrict the public endpoint to explicit operator +or CI egress CIDRs: + +```hcl +eks_enable_public_access = true +eks_public_access_cidrs = ["203.0.113.10/32"] +``` + +For private-only production applies, run Terraform from a VPC-connected +environment and set: + +```hcl +eks_enable_public_access = false +``` + +## EKS Access Entries + +The cluster creator keeps bootstrap admin access. Add explicit long-lived human +or CI access through `eks_access_entries`; for example: + +```hcl +eks_access_entries = { + platform_admin = { + principal_arn = "arn:aws:iam::123456789012:role/PlatformAdmin" + policy_associations = { + cluster_admin = { + policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy" + access_scope = { + type = "cluster" + } + } + } + } + + support_viewer = { + principal_arn = "arn:aws:iam::123456789012:role/BraintrustSupportViewer" + policy_associations = { + braintrust_logs = { + policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy" + access_scope = { + type = "namespace" + namespaces = ["braintrust"] + } + } + } + } + + ci_deployer = { + principal_arn = "arn:aws:iam::123456789012:role/BraintrustDeploy" + policy_associations = { + braintrust_deploy = { + policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy" + access_scope = { + type = "namespace" + namespaces = ["braintrust"] + } + } + } + } +} +``` + +Adjust `namespaces` if you change `eks_namespace`. + +## Apply + +```bash +terraform init +terraform apply +``` + +The example sets conservative Brainstore reader / fast-reader / writer defaults so +the dedicated Auto Mode NodePools have allocatable headroom on their fixed +`4xlarge` / `8xlarge` instance sizes. Override `eks_brainstore_reader_instance_sizes` +or `eks_brainstore_writer_instance_sizes` if you need larger reader or writer nodes. +If you increase Brainstore replicas or +requests, wait for the dedicated Brainstore pods to become `Running` before +registering the API URL. + +Check rollout status first: + +```bash +$(terraform output -raw connect_to_cluster) +kubectl get pods -n "$(terraform output -raw eks_namespace)" -w +``` + +After the API and Brainstore pods are ready, register the API URL in the Braintrust dashboard: + +```bash +terraform output -raw api_url +``` + +This example leaves `eks_enable_cloudfront_nlb_ingress = true`, so Terraform +also provisions the module-managed CloudFront + private NLB ingress path. + +For production public deployments, create an AWS WAF Web ACL with the managed +rules and rate limits that match your environment, then attach it to the +module-managed CloudFront distribution: + +```hcl +waf_acl_id = "arn:aws:wafv2:us-east-1:123456789012:global/webacl/braintrust/..." +``` + +If you need Kubernetes NetworkPolicy enforcement, add the VPC CNI network policy +configuration and NetworkPolicy manifests in your environment. This example +does not configure those policies for you. + +## Custom Domain + +To use your own hostname, provide: + +- `custom_domain` +- `custom_certificate_arn` + +The ACM certificate must live in `us-east-1`. After apply, create a DNS alias +to: + +```bash +terraform output -raw cloudfront_distribution_domain_name +terraform output -raw cloudfront_distribution_hosted_zone_id +``` + +## Connect To The Cluster + +```bash +$(terraform output -raw connect_to_cluster) +kubectl get pods -n "$(terraform output -raw eks_namespace)" +``` diff --git a/examples/eks-auto-mode/main.tf b/examples/eks-auto-mode/main.tf new file mode 100644 index 00000000..46964ac0 --- /dev/null +++ b/examples/eks-auto-mode/main.tf @@ -0,0 +1,108 @@ +# tflint-ignore-file: terraform_module_pinned_source + +# EKS Auto Mode deployment: +# - EKS Auto Mode manages system and API services compute, load balancing, and block storage +# automatically via its built-in system and general-purpose NodePools. +# - Brainstore reader, fast-reader, and writer pods run on custom Karpenter NodePools backed by +# NVMe-equipped Graviton instances (c7gd/c8gd). EKS Auto Mode provisions these nodes when the +# pods are scheduled; no managed node groups are created for Brainstore. +# - The AWS Load Balancer Controller is deployed to adopt the pre-created internal NLB. + +module "braintrust" { + source = "github.com/braintrustdata/terraform-braintrust-data-plane" + + deployment_name = var.deployment_name + braintrust_org_name = var.braintrust_org_name + brainstore_license_key = var.brainstore_license_key + custom_tags = var.custom_tags + + use_deployment_mode_eks = true + create_eks_cluster = true + enable_quarantine_vpc = false + + # EKS Auto Mode: EKS manages system and API services compute automatically. + eks_use_auto_mode = true + eks_enable_cloudfront_nlb_ingress = true + eks_enable_private_access = true + eks_enable_public_access = var.eks_enable_public_access + eks_public_access_cidrs = var.eks_public_access_cidrs + eks_access_entries = var.eks_access_entries + + custom_domain = var.custom_domain + custom_certificate_arn = var.custom_certificate_arn + waf_acl_id = var.waf_acl_id + cloudfront_price_class = var.cloudfront_price_class + + eks_namespace = var.eks_namespace + + kubernetes_version = var.kubernetes_version + + postgres_instance_type = var.postgres_instance_type + postgres_storage_size = var.postgres_storage_size + postgres_max_storage_size = var.postgres_max_storage_size + postgres_storage_type = "gp3" + postgres_storage_iops = var.postgres_storage_iops + postgres_storage_throughput = var.postgres_storage_throughput + + redis_instance_type = var.redis_instance_type + + brainstore_wal_footer_version = var.brainstore_wal_footer_version + skip_pg_for_brainstore_objects = var.skip_pg_for_brainstore_objects +} + +module "braintrust_deploy" { + source = "github.com/braintrustdata/terraform-braintrust-data-plane//modules/eks-deploy" + + providers = { + kubernetes = kubernetes + helm = helm + } + + deployment_name = var.deployment_name + custom_tags = var.custom_tags + braintrust_org_name = var.braintrust_org_name + namespace = module.braintrust.eks_namespace + + cluster_name = module.braintrust.eks_cluster_name + vpc_id = module.braintrust.main_vpc_id + nlb_security_group_id = module.braintrust.eks_nlb_security_group_id + nlb_name = module.braintrust.eks_nlb_name + + api_handler_role_arn = module.braintrust.eks_braintrust_api_role_arn + brainstore_iam_role_arn = module.braintrust.eks_brainstore_role_arn + api_service_account_name = "braintrust-api" + brainstore_service_account_name = "brainstore" + + brainstore_bucket_name = module.braintrust.brainstore_s3_bucket_name + response_bucket_name = module.braintrust.lambda_responses_s3_bucket_name + code_bundle_bucket_name = module.braintrust.code_bundle_s3_bucket_name + + postgres_host = module.braintrust.postgres_database_address + postgres_port = module.braintrust.postgres_database_port + postgres_username = module.braintrust.postgres_database_username + postgres_password = module.braintrust.postgres_database_password + redis_host = module.braintrust.redis_endpoint + redis_port = module.braintrust.redis_port + + brainstore_license_key = var.brainstore_license_key + function_secret_key = module.braintrust.function_tools_secret_key + brainstore_wal_footer_version = var.brainstore_wal_footer_version + skip_pg_for_brainstore_objects = var.skip_pg_for_brainstore_objects + + manage_braintrust_helm_release = true + helm_chart_version = var.eks_helm_chart_version + + api_helm = var.eks_api_helm + brainstore_reader_helm = var.eks_brainstore_reader_helm + brainstore_fastreader_helm = var.eks_brainstore_fastreader_helm + brainstore_writer_helm = var.eks_brainstore_writer_helm + helm_chart_extra_values = var.eks_helm_chart_extra_values + prepare_for_destroy = var.prepare_for_destroy + use_auto_mode = true + node_role_name = module.braintrust.eks_node_group_iam_role_name + brainstore_instance_families = var.eks_brainstore_instance_families + brainstore_reader_instance_sizes = var.eks_brainstore_reader_instance_sizes + brainstore_writer_instance_sizes = var.eks_brainstore_writer_instance_sizes + + depends_on = [module.braintrust] +} diff --git a/examples/eks-auto-mode/outputs.tf b/examples/eks-auto-mode/outputs.tf new file mode 100644 index 00000000..d7642dc2 --- /dev/null +++ b/examples/eks-auto-mode/outputs.tf @@ -0,0 +1,54 @@ +output "api_url" { + value = module.braintrust.api_url + description = "Braintrust API URL to register in the Braintrust dashboard." +} + +output "connect_to_cluster" { + value = "aws eks update-kubeconfig --region ${data.aws_region.current.region} --name ${module.braintrust.eks_cluster_name}" + description = "Command to configure kubectl for the EKS cluster." +} + +output "eks_cluster_name" { + value = module.braintrust.eks_cluster_name + description = "EKS cluster name." +} + +output "eks_namespace" { + value = module.braintrust.eks_namespace + description = "Kubernetes namespace where Braintrust is deployed." +} + +output "eks_braintrust_helm_release_managed" { + value = module.braintrust_deploy.braintrust_helm_release_managed + description = "True in this example because Terraform manages the Braintrust Helm release." +} + +output "eks_braintrust_helm_values" { + value = module.braintrust_deploy.braintrust_helm_values_yaml + description = "Generated multi-document Helm values passed to the Terraform-managed Braintrust Helm release." +} + +output "braintrust_release_name" { + value = module.braintrust_deploy.braintrust_release_name + description = "Helm release name used for the Terraform-managed Braintrust application deployment." +} + +output "cloudfront_distribution_domain_name" { + value = module.braintrust.cloudfront_distribution_domain_name + description = "CloudFront distribution domain name." +} + +output "cloudfront_distribution_hosted_zone_id" { + value = module.braintrust.cloudfront_distribution_hosted_zone_id + description = "CloudFront hosted zone ID for Route53 alias records." +} + +output "eks_nlb_name" { + value = module.braintrust.eks_nlb_name + description = "Internal NLB name adopted by the Braintrust API Kubernetes service." +} + +output "postgres_database_identifier" { + value = module.braintrust.postgres_database_identifier + description = "RDS instance identifier." +} diff --git a/examples/eks-auto-mode/provider.tf b/examples/eks-auto-mode/provider.tf new file mode 100644 index 00000000..f489eb57 --- /dev/null +++ b/examples/eks-auto-mode/provider.tf @@ -0,0 +1,33 @@ +# provider "aws" { +# region = "" +# +# # Optional but recommended. +# # profile = "" +# # allowed_account_ids = [""] +# } + +data "aws_region" "current" {} + +provider "kubernetes" { + host = module.braintrust.eks_cluster_endpoint + cluster_ca_certificate = base64decode(module.braintrust.eks_cluster_certificate_authority_data) + + exec { + api_version = "client.authentication.k8s.io/v1beta1" + command = "aws" + args = ["eks", "get-token", "--cluster-name", module.braintrust.eks_cluster_name, "--region", data.aws_region.current.region] + } +} + +provider "helm" { + kubernetes { + host = module.braintrust.eks_cluster_endpoint + cluster_ca_certificate = base64decode(module.braintrust.eks_cluster_certificate_authority_data) + + exec { + api_version = "client.authentication.k8s.io/v1beta1" + command = "aws" + args = ["eks", "get-token", "--cluster-name", module.braintrust.eks_cluster_name, "--region", data.aws_region.current.region] + } + } +} diff --git a/examples/eks-auto-mode/terraform.tf b/examples/eks-auto-mode/terraform.tf new file mode 100644 index 00000000..99c96c56 --- /dev/null +++ b/examples/eks-auto-mode/terraform.tf @@ -0,0 +1,10 @@ +# Configure remote state here, for example: +# +# terraform { +# backend "s3" { +# bucket = "" +# key = "braintrust-eks-auto-mode/terraform.tfstate" +# region = "" +# use_lockfile = true +# } +# } diff --git a/examples/eks-auto-mode/variables.tf b/examples/eks-auto-mode/variables.tf new file mode 100644 index 00000000..1419fe5a --- /dev/null +++ b/examples/eks-auto-mode/variables.tf @@ -0,0 +1,276 @@ +variable "deployment_name" { + description = "Name of this Braintrust deployment. Lowercase letters, numbers, and hyphens only. Do not change after initial deployment." + type = string + default = "braintrust" +} + +variable "braintrust_org_name" { + description = "Your organization name in Braintrust." + type = string +} + +variable "brainstore_license_key" { + description = "Brainstore license key from the Braintrust UI." + type = string + sensitive = true +} + +variable "custom_tags" { + description = "Additional AWS tags to apply to resources created by this example." + type = map(string) + default = {} +} + +variable "custom_domain" { + description = "Optional custom API hostname for CloudFront, e.g. braintrust.example.com. Leave null to use the CloudFront domain." + type = string + default = null +} + +variable "custom_certificate_arn" { + description = "ACM certificate ARN in us-east-1 for custom_domain. Required when custom_domain is set." + type = string + default = null +} + +variable "waf_acl_id" { + description = "Optional WAF Web ACL ARN/ID to attach to the CloudFront distribution." + type = string + default = null +} + +variable "cloudfront_price_class" { + description = "CloudFront price class for the Braintrust API edge." + type = string + default = "PriceClass_100" +} + +variable "eks_namespace" { + description = "Kubernetes namespace where Braintrust workloads are deployed." + type = string + default = "braintrust" +} + +variable "eks_helm_chart_version" { + description = "Version of the Braintrust Helm chart to deploy." + type = string +} + +variable "kubernetes_version" { + description = "Kubernetes version for the EKS cluster. EKS Auto Mode requires >= 1.29." + type = string + default = "1.31" +} + +variable "eks_enable_public_access" { + description = "Whether the EKS public API server endpoint is enabled. Set false only when Terraform runs from the VPC or a connected network." + type = bool + default = true +} + +variable "eks_public_access_cidrs" { + description = "CIDR blocks allowed to reach the EKS public API server endpoint. Defaults to 0.0.0.0/0 for compatibility; restrict this to explicit operator or CI egress CIDRs for production." + type = list(string) + default = ["0.0.0.0/0"] +} + +variable "eks_access_entries" { + description = "Additional EKS access entries for human operators or CI roles." + type = map(object({ + principal_arn = string + type = optional(string, "STANDARD") + kubernetes_groups = optional(list(string)) + user_name = optional(string) + policy_associations = optional(map(object({ + policy_arn = string + access_scope = object({ + type = string + namespaces = optional(list(string), []) + }) + })), {}) + })) + default = {} + + validation { + condition = alltrue(flatten([ + for _, entry in var.eks_access_entries : [ + for _, policy_association in entry.policy_associations : ( + contains(["cluster", "namespace"], policy_association.access_scope.type) && + ( + ( + policy_association.access_scope.type == "cluster" && + length(policy_association.access_scope.namespaces) == 0 + ) || + ( + policy_association.access_scope.type == "namespace" && + length(policy_association.access_scope.namespaces) > 0 + ) + ) + ) + ] + ])) + error_message = "Each EKS access policy association must use access_scope.type of cluster with no namespaces, or namespace with at least one namespace." + } +} + +variable "postgres_instance_type" { + description = "RDS instance type." + type = string + default = "db.r8g.2xlarge" +} + +variable "postgres_storage_size" { + description = "Initial RDS allocated storage in GB." + type = number + default = 1000 +} + +variable "postgres_max_storage_size" { + description = "Maximum RDS storage autoscaling limit in GB." + type = number + default = 10000 +} + +variable "postgres_storage_iops" { + description = "Provisioned gp3 IOPS for RDS." + type = number + default = 15000 +} + +variable "postgres_storage_throughput" { + description = "Provisioned gp3 throughput for RDS in MiB/s." + type = number + default = 500 +} + +variable "redis_instance_type" { + description = "ElastiCache node type for Redis." + type = string + default = "cache.t4g.medium" +} + +variable "eks_brainstore_instance_families" { + description = "NVMe-backed Graviton instance families for Brainstore Karpenter NodePools. Must support local NVMe storage (c7gd, c8gd)." + type = list(string) + default = ["c7gd", "c8gd"] +} + +variable "eks_brainstore_reader_instance_sizes" { + description = "EC2 instance sizes allowed for the Brainstore reader and fast-reader Auto Mode NodePool." + type = list(string) + default = ["4xlarge"] +} + +variable "eks_brainstore_writer_instance_sizes" { + description = "EC2 instance sizes allowed for the Brainstore writer Auto Mode NodePool." + type = list(string) + default = ["8xlarge"] +} + +variable "brainstore_wal_footer_version" { + description = "WAL footer version for Brainstore. Only change when instructed by Braintrust." + type = string + default = "v3" +} + +variable "skip_pg_for_brainstore_objects" { + description = "Controls which object types bypass PostgreSQL. This is a one-way migration setting." + type = string + default = "all" +} + +variable "eks_api_helm" { + type = object({ + replicas = optional(number) + resources = optional(object({ + requests = object({ cpu = string, memory = string }) + limits = object({ cpu = string, memory = string }) + })) + }) + default = {} + description = "Optional API replica/resource overrides for the Braintrust Helm chart." +} + +variable "eks_brainstore_reader_helm" { + type = object({ + replicas = optional(number) + resources = optional(object({ + requests = object({ cpu = string, memory = string }) + limits = object({ cpu = string, memory = string }) + })) + }) + default = { + replicas = 1 + resources = { + requests = { + cpu = "14" + memory = "26Gi" + } + limits = { + cpu = "14" + memory = "26Gi" + } + } + } + description = "Brainstore reader overrides for the Auto Mode example. Defaults leave allocatable headroom on the example's 4xlarge reader nodes; override if you need different sizing." +} + +variable "eks_brainstore_fastreader_helm" { + type = object({ + replicas = optional(number) + resources = optional(object({ + requests = object({ cpu = string, memory = string }) + limits = object({ cpu = string, memory = string }) + })) + }) + default = { + replicas = 1 + resources = { + requests = { + cpu = "14" + memory = "26Gi" + } + limits = { + cpu = "14" + memory = "26Gi" + } + } + } + description = "Brainstore fast-reader overrides for the Auto Mode example. Defaults leave allocatable headroom on the example's 4xlarge reader nodes; override if you need different sizing." +} + +variable "eks_brainstore_writer_helm" { + type = object({ + replicas = optional(number) + resources = optional(object({ + requests = object({ cpu = string, memory = string }) + limits = object({ cpu = string, memory = string }) + })) + }) + default = { + replicas = 1 + resources = { + requests = { + cpu = "30" + memory = "58Gi" + } + limits = { + cpu = "30" + memory = "58Gi" + } + } + } + description = "Brainstore writer overrides for the Auto Mode example. Defaults leave allocatable headroom on the example's 8xlarge writer nodes; override if you need different sizing." +} + +variable "eks_helm_chart_extra_values" { + description = "Raw YAML Helm override appended after the generated EKS values." + type = string + default = "" +} + +variable "prepare_for_destroy" { + description = "Apply with true before terraform destroy to shorten NLB target deregistration and reduce LB controller finalizer stalls." + type = bool + default = false +} diff --git a/examples/eks-auto-mode/versions.tf b/examples/eks-auto-mode/versions.tf new file mode 100644 index 00000000..88d6eb0f --- /dev/null +++ b/examples/eks-auto-mode/versions.tf @@ -0,0 +1,18 @@ +terraform { + required_version = ">= 1.10.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = "~> 2.0" + } + helm = { + source = "hashicorp/helm" + version = "~> 2.0" + } + } +} diff --git a/examples/eks-terraform-managed/README.md b/examples/eks-terraform-managed/README.md new file mode 100644 index 00000000..3d77c30b --- /dev/null +++ b/examples/eks-terraform-managed/README.md @@ -0,0 +1,214 @@ +# Braintrust Data Plane On Terraform-Managed EKS + +This example creates the Braintrust AWS data plane and an EKS cluster with +Terraform-managed node groups, but leaves the Braintrust application install to +you. + +Terraform creates: + +- the Braintrust AWS data plane resources +- the EKS cluster +- the managed node groups for system, services, and Brainstore workloads +- the IAM roles, buckets, database, Redis instance, and helper outputs you need + for a manual Helm install + +Terraform does not install the Braintrust Helm release, the Cluster Autoscaler, +or an ingress controller in this example. + +## What You Still Need + +- A cluster autoscaler strategy if you want the EKS node groups to scale beyond + their initial desired sizes. The example creates managed node groups with + min/max bounds, but it does not install Cluster Autoscaler or Karpenter for + you. +- An ingress path of your choice. This example defaults to + `eks_enable_cloudfront_nlb_ingress = false`, so you are expected to choose + and install your own ingress or service exposure model. + +Common choices: + +- an AWS Load Balancer Controller plus `Service` or `Ingress` +- an Istio gateway with the chart's `virtualService` settings +- any internal platform-specific ingress controller your team already uses + +If you want the module to provision the bundled CloudFront + private NLB edge +instead, set: + +```hcl +eks_enable_cloudfront_nlb_ingress = true +``` + +When the bundled ingress is enabled, you can also set: + +- `custom_domain` +- `custom_certificate_arn` + +The ACM certificate must live in `us-east-1`. + +## Required Inputs + +- `braintrust_org_name` +- `brainstore_license_key` + +## EKS API Endpoint Access + +This example creates the EKS infrastructure but does not install the Braintrust +Helm release. The Terraform infrastructure apply only needs AWS API access, but +future `kubectl` and `helm` commands must be able to reach the EKS API endpoint. + +For a laptop or public CI bootstrap, keep private access enabled. The example +defaults `eks_public_access_cidrs` to `["0.0.0.0/0"]` for compatibility, but +production deployments should restrict the public endpoint to explicit operator +or CI egress CIDRs: + +```hcl +eks_enable_public_access = true +eks_public_access_cidrs = ["203.0.113.10/32"] +``` + +For private-only clusters, run `kubectl` and Helm from the VPC or a connected +network, such as VPN, Direct Connect, Transit Gateway, a VPC-attached CI runner, +or an SSM-reached admin host, and set: + +```hcl +eks_enable_public_access = false +``` + +## EKS Access Entries + +The cluster creator keeps bootstrap admin access. Add explicit long-lived human +or CI access through `eks_access_entries`; for example: + +```hcl +eks_access_entries = { + platform_admin = { + principal_arn = "arn:aws:iam::123456789012:role/PlatformAdmin" + policy_associations = { + cluster_admin = { + policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy" + access_scope = { + type = "cluster" + } + } + } + } + + support_viewer = { + principal_arn = "arn:aws:iam::123456789012:role/BraintrustSupportViewer" + policy_associations = { + braintrust_logs = { + policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy" + access_scope = { + type = "namespace" + namespaces = ["braintrust"] + } + } + } + } + + ci_deployer = { + principal_arn = "arn:aws:iam::123456789012:role/BraintrustDeploy" + policy_associations = { + braintrust_deploy = { + policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy" + access_scope = { + type = "namespace" + namespaces = ["braintrust"] + } + } + } + } +} +``` + +Adjust `namespaces` if you change `eks_namespace`. + +If you need Kubernetes NetworkPolicy enforcement, add the VPC CNI network policy +configuration and NetworkPolicy manifests in your environment. This example +does not configure those policies for you. + +## Apply + +```bash +terraform init +terraform apply +``` + +## Connect To The Cluster + +```bash +terraform output -raw connect_to_cluster | bash +kubectl get pods -n "$(terraform output -raw eks_namespace)" +``` + +## Create The Kubernetes Secret + +This example exposes a ready-to-run secret command, but it intentionally leaves +the Brainstore license as a placeholder so you do not leak the actual license +via outputs. + +```bash +terraform output -raw braintrust_secrets_command +``` + +Replace `` with your real Brainstore license key, then run +the command. + +## Write The Base Helm Values File + +```bash +terraform output -raw braintrust_write_generated_values_command | bash +``` + +This writes `braintrust-generated-values.yaml`, which includes: + +- the Braintrust org and namespace +- S3 bucket names +- API and Brainstore IAM role ARNs +- node selectors and tolerations that match the dedicated Terraform-managed + node groups + +Create a separate `values.yaml` for your environment-specific overrides, +especially ingress and exposure settings. + +Examples of what typically goes into `values.yaml`: + +- your ingress or gateway configuration +- any custom hostname configuration +- service type overrides if you are not using `ClusterIP` +- resource overrides if you want something different from chart defaults + +## Install The Public Braintrust Helm Chart + +```bash +terraform output -raw braintrust_public_helm_command +``` + +That prints a command like: + +```bash +helm upgrade --install braintrust \ + oci://public.ecr.aws/braintrust/helm/braintrust \ + --namespace braintrust \ + --create-namespace \ + --values ./braintrust-generated-values.yaml \ + --values ./values.yaml +``` + +## Useful Outputs + +- `connect_to_cluster`: configure `kubectl` for the new EKS cluster +- `postgres_database_secret_arn`: Secrets Manager ARN for the database creds +- `postgres_database_address`: PostgreSQL hostname +- `postgres_database_port`: PostgreSQL port +- `redis_endpoint`: Redis hostname +- `redis_port`: Redis port +- `function_tools_secret_key`: function tools secret key +- `braintrust_secrets_command`: ready-to-edit secret creation command with a + `` placeholder +- `braintrust_generated_values_yaml`: generated base Helm values +- `braintrust_write_generated_values_command`: writes + `braintrust-generated-values.yaml` +- `braintrust_public_helm_command`: public OCI Helm install command +- `api_url`: stays `null` until you enable bundled ingress or add your own + ingress path diff --git a/examples/eks-terraform-managed/main.tf b/examples/eks-terraform-managed/main.tf new file mode 100644 index 00000000..f154bde4 --- /dev/null +++ b/examples/eks-terraform-managed/main.tf @@ -0,0 +1,88 @@ +# tflint-ignore-file: terraform_module_pinned_source + +# Opinionated Terraform-managed EKS example: +# - creates the Braintrust AWS data plane primitives +# - creates an EKS cluster with Terraform-managed node groups +# - leaves Braintrust Helm installation to a follow-up manual step +# - defaults to bring-your-own ingress, with an optional toggle for bundled CloudFront + private NLB + +module "braintrust" { + source = "github.com/braintrustdata/terraform-braintrust-data-plane" + + deployment_name = var.deployment_name + braintrust_org_name = var.braintrust_org_name + brainstore_license_key = var.brainstore_license_key + custom_tags = var.custom_tags + + use_deployment_mode_eks = true + create_eks_cluster = true + enable_quarantine_vpc = false + + eks_use_auto_mode = false + eks_enable_cloudfront_nlb_ingress = var.eks_enable_cloudfront_nlb_ingress + eks_enable_private_access = true + eks_enable_public_access = var.eks_enable_public_access + eks_public_access_cidrs = var.eks_public_access_cidrs + eks_access_entries = var.eks_access_entries + + custom_domain = var.custom_domain + custom_certificate_arn = var.custom_certificate_arn + waf_acl_id = var.waf_acl_id + cloudfront_price_class = var.cloudfront_price_class + + eks_namespace = var.eks_namespace + + kubernetes_version = var.kubernetes_version + eks_enable_node_ssm = var.eks_enable_node_ssm + eks_node_group_ami_type = var.eks_node_group_ami_type + eks_brainstore_node_group_ami_type = var.eks_brainstore_node_group_ami_type + + eks_system_node_group_instance_type = var.eks_system_node_group_instance_type + eks_system_node_group_desired_size = var.eks_system_node_group_desired_size + eks_system_node_group_min_size = var.eks_system_node_group_min_size + eks_system_node_group_max_size = var.eks_system_node_group_max_size + eks_system_node_group_disk_size = var.eks_system_node_group_disk_size + + eks_services_node_group_instance_type = var.eks_services_node_group_instance_type + eks_services_node_group_desired_size = var.eks_services_node_group_desired_size + eks_services_node_group_min_size = var.eks_services_node_group_min_size + eks_services_node_group_max_size = var.eks_services_node_group_max_size + eks_services_node_group_disk_size = var.eks_services_node_group_disk_size + + eks_brainstore_reader_node_group_instance_type = var.eks_brainstore_reader_node_group_instance_type + eks_brainstore_reader_node_group_desired_size = var.eks_brainstore_reader_node_group_desired_size + eks_brainstore_reader_node_group_min_size = var.eks_brainstore_reader_node_group_min_size + eks_brainstore_reader_node_group_max_size = var.eks_brainstore_reader_node_group_max_size + + eks_brainstore_writer_node_group_instance_type = var.eks_brainstore_writer_node_group_instance_type + eks_brainstore_writer_node_group_desired_size = var.eks_brainstore_writer_node_group_desired_size + eks_brainstore_writer_node_group_min_size = var.eks_brainstore_writer_node_group_min_size + eks_brainstore_writer_node_group_max_size = var.eks_brainstore_writer_node_group_max_size + + eks_enable_services_spot_node_group = var.eks_enable_services_spot_node_group + eks_services_spot_node_group_instance_types = var.eks_services_spot_node_group_instance_types + eks_services_spot_node_group_min_size = var.eks_services_spot_node_group_min_size + eks_services_spot_node_group_max_size = var.eks_services_spot_node_group_max_size + + eks_enable_brainstore_spot_node_group = var.eks_enable_brainstore_spot_node_group + eks_brainstore_spot_node_group_instance_types = var.eks_brainstore_spot_node_group_instance_types + eks_brainstore_spot_node_group_min_size = var.eks_brainstore_spot_node_group_min_size + eks_brainstore_spot_node_group_max_size = var.eks_brainstore_spot_node_group_max_size + + eks_enable_brainstore_writer_spot_node_group = var.eks_enable_brainstore_writer_spot_node_group + eks_brainstore_writer_spot_node_group_instance_types = var.eks_brainstore_writer_spot_node_group_instance_types + eks_brainstore_writer_spot_node_group_min_size = var.eks_brainstore_writer_spot_node_group_min_size + eks_brainstore_writer_spot_node_group_max_size = var.eks_brainstore_writer_spot_node_group_max_size + + postgres_instance_type = var.postgres_instance_type + postgres_storage_size = var.postgres_storage_size + postgres_max_storage_size = var.postgres_max_storage_size + postgres_storage_type = "gp3" + postgres_storage_iops = var.postgres_storage_iops + postgres_storage_throughput = var.postgres_storage_throughput + postgres_version = "15" + postgres_auto_minor_version_upgrade = true + + redis_instance_type = var.redis_instance_type + redis_version = "7.0" +} diff --git a/examples/eks-terraform-managed/outputs.tf b/examples/eks-terraform-managed/outputs.tf new file mode 100644 index 00000000..582222a6 --- /dev/null +++ b/examples/eks-terraform-managed/outputs.tf @@ -0,0 +1,195 @@ +output "api_url" { + value = module.braintrust.api_url + description = "Braintrust API URL to register in the Braintrust dashboard. Null until you enable bundled ingress or add your own ingress path." +} + +output "connect_to_cluster" { + value = "aws eks update-kubeconfig --region ${data.aws_region.current.region} --name ${module.braintrust.eks_cluster_name}" + description = "Command to configure kubectl for the EKS cluster." +} + +output "eks_cluster_name" { + value = module.braintrust.eks_cluster_name + description = "EKS cluster name." +} + +output "eks_namespace" { + value = module.braintrust.eks_namespace + description = "Kubernetes namespace where Braintrust is deployed." +} + +output "cloudfront_distribution_domain_name" { + value = module.braintrust.cloudfront_distribution_domain_name + description = "CloudFront distribution domain name. Null when bundled ingress is disabled." +} + +output "cloudfront_distribution_hosted_zone_id" { + value = module.braintrust.cloudfront_distribution_hosted_zone_id + description = "CloudFront hosted zone ID for Route53 alias records. Null when bundled ingress is disabled." +} + +output "eks_nlb_name" { + value = module.braintrust.eks_nlb_name + description = "Internal NLB name adopted by the Braintrust API Kubernetes service. Null when bundled ingress is disabled." +} + +output "postgres_database_identifier" { + value = module.braintrust.postgres_database_identifier + description = "RDS instance identifier." +} + +output "postgres_database_secret_arn" { + value = module.braintrust.postgres_database_secret_arn + description = "ARN of the Secrets Manager secret containing PostgreSQL credentials." +} + +output "postgres_database_address" { + value = module.braintrust.postgres_database_address + description = "Hostname of the PostgreSQL database." +} + +output "postgres_database_port" { + value = module.braintrust.postgres_database_port + description = "Port of the PostgreSQL database." +} + +output "redis_endpoint" { + value = module.braintrust.redis_endpoint + description = "Hostname of the ElastiCache Redis instance." +} + +output "redis_port" { + value = module.braintrust.redis_port + description = "Port of the ElastiCache Redis instance." +} + +output "function_tools_secret_key" { + value = module.braintrust.function_tools_secret_key + description = "Function secret key used by Braintrust application components." + sensitive = true +} + +output "braintrust_secrets_command" { + value = <<-EOT +DB_SECRET_JSON="$(aws secretsmanager get-secret-value \ + --secret-id $(terraform output -raw postgres_database_secret_arn) \ + --region ${data.aws_region.current.region} \ + --query SecretString \ + --output text)" + +DB_USER="$(jq -r '.username' <<<"$DB_SECRET_JSON")" +DB_PASS="$(jq -r '.password' <<<"$DB_SECRET_JSON")" + +kubectl create secret generic braintrust-secrets \ + -n $(terraform output -raw eks_namespace) \ + --from-literal=PG_URL="postgresql://$${DB_USER}:$${DB_PASS}@$(terraform output -raw postgres_database_address):$(terraform output -raw postgres_database_port)/postgres?sslmode=require" \ + --from-literal=REDIS_URL="redis://$(terraform output -raw redis_endpoint):$(terraform output -raw redis_port)" \ + --from-literal=BRAINSTORE_LICENSE_KEY="" \ + --from-literal=FUNCTION_SECRET_KEY="$(terraform output -raw function_tools_secret_key)" \ + --dry-run=client -o yaml | kubectl apply -f - +EOT + description = "Command to create or update the braintrust-secrets secret. Replace with a real license key before running." +} + +output "braintrust_generated_values_yaml" { + value = yamlencode({ + global = { + orgName = var.braintrust_org_name + namespace = module.braintrust.eks_namespace + createNamespace = false + } + + cloud = "aws" + + objectStorage = { + aws = { + brainstoreBucket = module.braintrust.brainstore_s3_bucket_name + responseBucket = module.braintrust.lambda_responses_s3_bucket_name + codeBundleBucket = module.braintrust.code_bundle_s3_bucket_name + } + } + + skipPgForBrainstoreObjects = "all" + brainstoreWalFooterVersion = "v3" + + api = { + service = { + type = "ClusterIP" + port = 8000 + portName = "http" + } + serviceAccount = { + name = "braintrust-api" + awsRoleArn = module.braintrust.eks_braintrust_api_role_arn + } + nodeSelector = { + role = "services" + } + tolerations = [ + { + key = "dedicated" + operator = "Equal" + value = "services" + effect = "NoSchedule" + } + ] + } + + brainstore = { + serviceAccount = { + name = "brainstore" + awsRoleArn = module.braintrust.eks_brainstore_role_arn + } + reader = { + nodeSelector = { + role = "brainstore-reader" + } + tolerations = [ + { + key = "dedicated" + operator = "Equal" + value = "brainstore" + effect = "NoSchedule" + } + ] + } + fastreader = { + nodeSelector = { + role = "brainstore-reader" + } + tolerations = [ + { + key = "dedicated" + operator = "Equal" + value = "brainstore" + effect = "NoSchedule" + } + ] + } + writer = { + nodeSelector = { + role = "brainstore-writer" + } + tolerations = [ + { + key = "dedicated" + operator = "Equal" + value = "brainstore" + effect = "NoSchedule" + } + ] + } + } + }) + description = "Generated base Helm values for the Braintrust chart. Add ingress-specific overrides in a separate values.yaml." +} + +output "braintrust_write_generated_values_command" { + value = "terraform output -raw braintrust_generated_values_yaml > ./braintrust-generated-values.yaml" + description = "Command to write the generated Helm values to braintrust-generated-values.yaml in this directory." +} + +output "braintrust_public_helm_command" { + value = "helm upgrade --install braintrust oci://public.ecr.aws/braintrust/helm/braintrust --namespace ${module.braintrust.eks_namespace} --create-namespace --values ./braintrust-generated-values.yaml --values ./values.yaml" + description = "Command to install or upgrade the public Braintrust Helm chart using the generated base values plus your own ingress-specific overrides." +} diff --git a/examples/eks-terraform-managed/provider.tf b/examples/eks-terraform-managed/provider.tf new file mode 100644 index 00000000..7e53c192 --- /dev/null +++ b/examples/eks-terraform-managed/provider.tf @@ -0,0 +1,9 @@ +# provider "aws" { +# region = "" +# +# # Optional but recommended. +# # profile = "" +# # allowed_account_ids = [""] +# } + +data "aws_region" "current" {} diff --git a/examples/eks-terraform-managed/terraform.tf b/examples/eks-terraform-managed/terraform.tf new file mode 100644 index 00000000..05bfaf0b --- /dev/null +++ b/examples/eks-terraform-managed/terraform.tf @@ -0,0 +1,10 @@ +# Configure remote state here, for example: +# +# terraform { +# backend "s3" { +# bucket = "" +# key = "braintrust-eks/terraform.tfstate" +# region = "" +# use_lockfile = true +# } +# } diff --git a/examples/eks-terraform-managed/variables.tf b/examples/eks-terraform-managed/variables.tf new file mode 100644 index 00000000..0d2d206b --- /dev/null +++ b/examples/eks-terraform-managed/variables.tf @@ -0,0 +1,361 @@ +variable "deployment_name" { + description = "Name of this Braintrust deployment. Lowercase letters, numbers, and hyphens only. Do not change after initial deployment." + type = string + default = "braintrust" +} + +variable "braintrust_org_name" { + description = "Your organization name in Braintrust." + type = string +} + +variable "brainstore_license_key" { + description = "Brainstore license key from the Braintrust UI." + type = string + sensitive = true +} + +variable "custom_tags" { + description = "Additional AWS tags to apply to resources created by this example." + type = map(string) + default = {} +} + +variable "custom_domain" { + description = "Optional custom API hostname for CloudFront, e.g. braintrust.example.com. Leave null to use the CloudFront domain." + type = string + default = null +} + +variable "custom_certificate_arn" { + description = "ACM certificate ARN in us-east-1 for custom_domain. Required when custom_domain is set." + type = string + default = null +} + +variable "waf_acl_id" { + description = "Optional WAF Web ACL ARN/ID to attach to the CloudFront distribution." + type = string + default = null +} + +variable "cloudfront_price_class" { + description = "CloudFront price class for the Braintrust API edge." + type = string + default = "PriceClass_100" +} + +variable "eks_enable_cloudfront_nlb_ingress" { + description = "When true, enables the bundled CloudFront + private NLB ingress. Leave false to bring your own ingress." + type = bool + default = false +} + +variable "eks_namespace" { + description = "Kubernetes namespace where Braintrust workloads are deployed." + type = string + default = "braintrust" +} + +variable "kubernetes_version" { + description = "Kubernetes version for the EKS cluster." + type = string + default = "1.31" +} + +variable "eks_enable_public_access" { + description = "Whether the EKS public API server endpoint is enabled. Set false when all kubectl/helm access comes from the VPC or a connected network." + type = bool + default = true +} + +variable "eks_public_access_cidrs" { + description = "CIDR blocks allowed to reach the EKS public API server endpoint. Defaults to 0.0.0.0/0 for compatibility; restrict this to explicit operator or CI egress CIDRs for production." + type = list(string) + default = ["0.0.0.0/0"] +} + +variable "eks_access_entries" { + description = "Additional EKS access entries for human operators or CI roles." + type = map(object({ + principal_arn = string + type = optional(string, "STANDARD") + kubernetes_groups = optional(list(string)) + user_name = optional(string) + policy_associations = optional(map(object({ + policy_arn = string + access_scope = object({ + type = string + namespaces = optional(list(string), []) + }) + })), {}) + })) + default = {} + + validation { + condition = alltrue(flatten([ + for _, entry in var.eks_access_entries : [ + for _, policy_association in entry.policy_associations : ( + contains(["cluster", "namespace"], policy_association.access_scope.type) && + ( + ( + policy_association.access_scope.type == "cluster" && + length(policy_association.access_scope.namespaces) == 0 + ) || + ( + policy_association.access_scope.type == "namespace" && + length(policy_association.access_scope.namespaces) > 0 + ) + ) + ) + ] + ])) + error_message = "Each EKS access policy association must use access_scope.type of cluster with no namespaces, or namespace with at least one namespace." + } +} + +variable "eks_enable_node_ssm" { + description = "Enable AWS Systems Manager Session Manager on EKS nodes for debugging." + type = bool + default = false +} + +variable "eks_node_group_ami_type" { + description = "AMI type for the system and services node groups." + type = string + default = "AL2023_x86_64_STANDARD" +} + +variable "eks_brainstore_node_group_ami_type" { + description = "AMI type for the Brainstore reader and writer node groups." + type = string + default = "AL2023_ARM_64_STANDARD" +} + +variable "eks_system_node_group_instance_type" { + description = "Instance type for the system node group." + type = string + default = "t3.medium" +} + +variable "eks_system_node_group_desired_size" { + description = "Desired node count for the system node group." + type = number + default = 2 +} + +variable "eks_system_node_group_min_size" { + description = "Minimum node count for the system node group." + type = number + default = 2 +} + +variable "eks_system_node_group_max_size" { + description = "Maximum node count for the system node group." + type = number + default = 4 +} + +variable "eks_system_node_group_disk_size" { + description = "Root EBS volume size in GB for the system node group." + type = number + default = 50 +} + +variable "eks_services_node_group_instance_type" { + description = "Instance type for the services node group." + type = string + default = "r6i.2xlarge" +} + +variable "eks_services_node_group_desired_size" { + description = "Desired node count for the services node group." + type = number + default = 2 +} + +variable "eks_services_node_group_min_size" { + description = "Minimum node count for the services node group." + type = number + default = 2 +} + +variable "eks_services_node_group_max_size" { + description = "Maximum node count for the services node group." + type = number + default = 10 +} + +variable "eks_services_node_group_disk_size" { + description = "Root EBS volume size in GB for the services node group." + type = number + default = 100 +} + +variable "eks_brainstore_reader_node_group_instance_type" { + description = "Instance type for the Brainstore reader node group. Must have local NVMe storage." + type = string + default = "c8gd.8xlarge" +} + +variable "eks_brainstore_reader_node_group_desired_size" { + description = "Desired node count for the Brainstore reader node group." + type = number + default = 2 +} + +variable "eks_brainstore_reader_node_group_min_size" { + description = "Minimum node count for the Brainstore reader node group." + type = number + default = 2 +} + +variable "eks_brainstore_reader_node_group_max_size" { + description = "Maximum node count for the Brainstore reader node group." + type = number + default = 10 +} + +variable "eks_brainstore_writer_node_group_instance_type" { + description = "Instance type for the Brainstore writer node group. Must have local NVMe storage." + type = string + default = "c8gd.16xlarge" +} + +variable "eks_brainstore_writer_node_group_desired_size" { + description = "Desired node count for the Brainstore writer node group." + type = number + default = 1 +} + +variable "eks_brainstore_writer_node_group_min_size" { + description = "Minimum node count for the Brainstore writer node group." + type = number + default = 1 +} + +variable "eks_brainstore_writer_node_group_max_size" { + description = "Maximum node count for the Brainstore writer node group." + type = number + default = 5 +} + +variable "eks_enable_services_spot_node_group" { + description = "Enable an additional spot-backed services node group for burst capacity." + type = bool + default = false +} + +variable "eks_services_spot_node_group_instance_types" { + description = "Instance types for the services spot node group." + type = list(string) + default = ["r6i.2xlarge", "r5.2xlarge", "r6a.2xlarge"] +} + +variable "eks_services_spot_node_group_min_size" { + description = "Minimum node count for the services spot node group." + type = number + default = 0 +} + +variable "eks_services_spot_node_group_max_size" { + description = "Maximum node count for the services spot node group." + type = number + default = 9 +} + +variable "eks_enable_brainstore_spot_node_group" { + description = "Enable an additional spot-backed Brainstore reader node group for burst capacity." + type = bool + default = false +} + +variable "eks_brainstore_spot_node_group_instance_types" { + description = "Instance types for the Brainstore reader spot node group. Must have local NVMe storage." + type = list(string) + default = ["c8gd.8xlarge", "c8gd.12xlarge", "c6gd.8xlarge"] +} + +variable "eks_brainstore_spot_node_group_min_size" { + description = "Minimum node count for the Brainstore reader spot node group." + type = number + default = 0 +} + +variable "eks_brainstore_spot_node_group_max_size" { + description = "Maximum node count for the Brainstore reader spot node group." + type = number + default = 9 +} + +variable "eks_enable_brainstore_writer_spot_node_group" { + description = "Enable an additional spot-backed Brainstore writer node group for burst capacity." + type = bool + default = false +} + +variable "eks_brainstore_writer_spot_node_group_instance_types" { + description = "Instance types for the Brainstore writer spot node group. Must have local NVMe storage." + type = list(string) + default = ["c8gd.16xlarge", "c7gd.16xlarge", "c6gd.16xlarge"] +} + +variable "eks_brainstore_writer_spot_node_group_min_size" { + description = "Minimum node count for the Brainstore writer spot node group." + type = number + default = 0 +} + +variable "eks_brainstore_writer_spot_node_group_max_size" { + description = "Maximum node count for the Brainstore writer spot node group." + type = number + default = 9 +} + +variable "postgres_instance_type" { + description = "RDS instance type." + type = string + default = "db.r8g.2xlarge" +} + +variable "postgres_storage_size" { + description = "Initial RDS allocated storage in GB." + type = number + default = 1000 +} + +variable "postgres_max_storage_size" { + description = "Maximum RDS storage autoscaling limit in GB." + type = number + default = 10000 +} + +variable "postgres_storage_iops" { + description = "Provisioned gp3 IOPS for RDS." + type = number + default = 15000 +} + +variable "postgres_storage_throughput" { + description = "Provisioned gp3 throughput for RDS in MiB/s." + type = number + default = 500 +} + +variable "redis_instance_type" { + description = "ElastiCache node type for Redis." + type = string + default = "cache.t4g.medium" +} + +variable "brainstore_wal_footer_version" { + description = "WAL footer version for Brainstore. Only change when instructed by Braintrust." + type = string + default = "v3" +} + +variable "skip_pg_for_brainstore_objects" { + description = "Controls which object types bypass PostgreSQL. This is a one-way migration setting." + type = string + default = "all" +} diff --git a/examples/eks-terraform-managed/versions.tf b/examples/eks-terraform-managed/versions.tf new file mode 100644 index 00000000..691becc5 --- /dev/null +++ b/examples/eks-terraform-managed/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.10.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + } +} diff --git a/main.tf b/main.tf index 4f1099e8..dcf345ea 100644 --- a/main.tf +++ b/main.tf @@ -16,12 +16,25 @@ locals { "EC2 Instance Connect Endpoint" = module.remote_support[0].instance_connect_endpoint_security_group_id } : {} + # EKS deployment mode — prefer the new flag name, but keep the deprecated alias working. + use_eks_deployment_mode = var.use_deployment_mode_eks || var.use_deployment_mode_external_eks + + # EKS deployment mode — true when either the cluster is created here or managed externally + use_eks = var.create_eks_cluster || local.use_eks_deployment_mode + + eks_namespace_resolved = coalesce(var.eks_namespace, "braintrust") + + # EKS cluster ARN — prefer the cluster this module created, fall back to externally provided ARN + eks_cluster_arn = var.create_eks_cluster ? module.eks_cluster[0].cluster_arn : var.existing_eks_cluster_arn + # VPC configuration - handle both created and existing VPCs main_vpc_id = var.create_vpc ? module.main_vpc[0].vpc_id : var.existing_vpc_id main_vpc_private_subnet_1_id = var.create_vpc ? module.main_vpc[0].private_subnet_1_id : var.existing_private_subnet_1_id main_vpc_private_subnet_2_id = var.create_vpc ? module.main_vpc[0].private_subnet_2_id : var.existing_private_subnet_2_id main_vpc_private_subnet_3_id = var.create_vpc ? module.main_vpc[0].private_subnet_3_id : var.existing_private_subnet_3_id main_vpc_public_subnet_1_id = var.create_vpc ? module.main_vpc[0].public_subnet_1_id : var.existing_public_subnet_1_id + main_vpc_public_subnet_2_id = var.create_vpc ? module.main_vpc[0].public_subnet_2_id : var.existing_public_subnet_2_id + main_vpc_public_subnet_3_id = var.create_vpc ? module.main_vpc[0].public_subnet_3_id : null # Quarantine VPC configuration - handle both created and existing VPCs create_quarantine_vpc = var.enable_quarantine_vpc && var.existing_quarantine_vpc_id == null @@ -48,6 +61,10 @@ module "main_vpc" { public_subnet_1_cidr = cidrsubnet(var.vpc_cidr, 3, 0) public_subnet_1_az = local.public_subnet_1_az + public_subnet_2_cidr = var.public_subnet_count >= 2 ? cidrsubnet(var.vpc_cidr, 3, 4) : null + public_subnet_2_az = local.public_subnet_2_az + public_subnet_3_cidr = var.public_subnet_count >= 3 ? cidrsubnet(var.vpc_cidr, 3, 5) : null + public_subnet_3_az = local.public_subnet_3_az private_subnet_1_cidr = cidrsubnet(var.vpc_cidr, 3, 1) private_subnet_1_az = local.private_subnet_1_az private_subnet_2_cidr = cidrsubnet(var.vpc_cidr, 3, 2) @@ -96,8 +113,9 @@ module "database" { "Brainstore" = module.services_common.brainstore_instance_security_group_id }, var.database_authorized_security_groups, + var.create_eks_cluster ? { "EKS Cluster" = module.eks_cluster[0].cluster_security_group_id } : {}, # This is a deprecated security group that will be removed in the future - !var.use_deployment_mode_external_eks ? { "Lambda Services" = module.services[0].lambda_security_group_id } : {} + !local.use_eks ? { "Lambda Services" = module.services[0].lambda_security_group_id } : {} ), local.bastion_security_group, ) @@ -129,8 +147,9 @@ module "redis" { "Brainstore" = module.services_common.brainstore_instance_security_group_id }, var.redis_authorized_security_groups, + var.create_eks_cluster ? { "EKS Cluster" = module.eks_cluster[0].cluster_security_group_id } : {}, # This is a deprecated security group that will be removed in the future - !var.use_deployment_mode_external_eks ? { "Lambda Services" = module.services[0].lambda_security_group_id } : {} + !local.use_eks ? { "Lambda Services" = module.services[0].lambda_security_group_id } : {} ), local.bastion_security_group, ) @@ -151,7 +170,7 @@ module "storage" { module "services" { source = "./modules/services" - count = !var.use_deployment_mode_external_eks ? 1 : 0 + count = !local.use_eks ? 1 : 0 deployment_name = var.deployment_name lambda_version_tag_override = var.lambda_version_tag_override @@ -280,12 +299,12 @@ module "gateway_ecs" { brainstore_license_key = var.brainstore_license_key enable_execute_command = var.gateway_enable_execute_command braintrust_app_url = var.gateway_braintrust_app_url - braintrust_api_url = var.use_deployment_mode_external_eks ? var.braintrust_api_url : module.ingress[0].api_url + braintrust_api_url = var.create_eks_cluster ? coalesce(module.eks_cluster[0].api_url, var.braintrust_api_url) : (local.use_eks ? var.braintrust_api_url : module.ingress[0].api_url) } module "ingress" { source = "./modules/ingress" - count = !var.use_deployment_mode_external_eks ? 1 : 0 + count = !local.use_eks ? 1 : 0 deployment_name = var.deployment_name custom_domain = var.custom_domain @@ -312,9 +331,11 @@ module "services_common" { brainstore_additional_policy_arns = var.brainstore_additional_policy_arns brainstore_enable_export = var.brainstore_enable_export permissions_boundary_arn = var.permissions_boundary_arn - eks_cluster_arn = var.existing_eks_cluster_arn - eks_namespace = var.eks_namespace - enable_eks_pod_identity = var.enable_eks_pod_identity + eks_cluster_arn = local.eks_cluster_arn + eks_cluster_oidc_issuer_url = var.create_eks_cluster ? module.eks_cluster[0].cluster_oidc_issuer_url : null + lookup_eks_cluster_oidc_issuer_url = !var.create_eks_cluster && var.existing_eks_cluster_arn != null + eks_namespace = local.eks_namespace_resolved + enable_eks_pod_identity = var.create_eks_cluster ? true : var.enable_eks_pod_identity enable_eks_irsa = var.enable_eks_irsa enable_brainstore_ec2_ssm = var.enable_brainstore_ec2_ssm custom_tags = var.custom_tags @@ -326,7 +347,7 @@ module "services_common" { module "brainstore" { source = "./modules/brainstore-ec2" - count = var.enable_brainstore && !var.use_deployment_mode_external_eks ? 1 : 0 + count = var.enable_brainstore && !local.use_eks ? 1 : 0 deployment_name = var.deployment_name instance_count = var.brainstore_instance_count @@ -366,7 +387,7 @@ module "brainstore" { "API" = module.services_common.api_security_group_id }, # This is a deprecated security group that will be removed in the future - !var.use_deployment_mode_external_eks ? { "Lambda Services" = module.services[0].lambda_security_group_id } : {} + !local.use_eks ? { "Lambda Services" = module.services[0].lambda_security_group_id } : {} ), local.bastion_security_group ) @@ -389,3 +410,91 @@ module "brainstore" { cache_file_size_writer = var.brainstore_cache_file_size_writer locks_s3_path = var.brainstore_locks_s3_path } + +module "eks_cluster" { + source = "./modules/eks-cluster" + count = var.create_eks_cluster ? 1 : 0 + + deployment_name = var.deployment_name + kms_key_arn = local.kms_key_arn + vpc_id = local.main_vpc_id + permissions_boundary_arn = var.permissions_boundary_arn + subnet_ids = [ + local.main_vpc_private_subnet_1_id, + local.main_vpc_private_subnet_2_id, + local.main_vpc_private_subnet_3_id + ] + + use_eks_auto_mode = var.eks_use_auto_mode + kubernetes_version = var.kubernetes_version + enable_private_access = var.eks_enable_private_access + enable_public_access = var.eks_enable_public_access + public_access_cidrs = var.eks_public_access_cidrs + eks_access_entries = var.eks_access_entries + + cluster_log_types = var.eks_cluster_log_types + cluster_log_retention_days = var.eks_cluster_log_retention_days + + enable_node_ssm = var.eks_enable_node_ssm + node_group_ami_type = var.eks_node_group_ami_type + brainstore_node_group_ami_type = var.eks_brainstore_node_group_ami_type + + system_node_group_instance_type = var.eks_system_node_group_instance_type + system_node_group_desired_size = var.eks_system_node_group_desired_size + system_node_group_min_size = var.eks_system_node_group_min_size + system_node_group_max_size = var.eks_system_node_group_max_size + system_node_group_disk_size = var.eks_system_node_group_disk_size + + services_node_group_instance_type = var.eks_services_node_group_instance_type + services_node_group_desired_size = var.eks_services_node_group_desired_size + services_node_group_min_size = var.eks_services_node_group_min_size + services_node_group_max_size = var.eks_services_node_group_max_size + services_node_group_disk_size = var.eks_services_node_group_disk_size + + brainstore_reader_node_group_instance_type = var.eks_brainstore_reader_node_group_instance_type + brainstore_reader_node_group_desired_size = var.eks_brainstore_reader_node_group_desired_size + brainstore_reader_node_group_min_size = var.eks_brainstore_reader_node_group_min_size + brainstore_reader_node_group_max_size = var.eks_brainstore_reader_node_group_max_size + + brainstore_writer_node_group_instance_type = var.eks_brainstore_writer_node_group_instance_type + brainstore_writer_node_group_desired_size = var.eks_brainstore_writer_node_group_desired_size + brainstore_writer_node_group_min_size = var.eks_brainstore_writer_node_group_min_size + brainstore_writer_node_group_max_size = var.eks_brainstore_writer_node_group_max_size + + enable_services_spot_node_group = var.eks_enable_services_spot_node_group + services_spot_node_group_instance_types = var.eks_services_spot_node_group_instance_types + services_spot_node_group_min_size = var.eks_services_spot_node_group_min_size + services_spot_node_group_max_size = var.eks_services_spot_node_group_max_size + + enable_brainstore_spot_node_group = var.eks_enable_brainstore_spot_node_group + brainstore_spot_node_group_instance_types = var.eks_brainstore_spot_node_group_instance_types + brainstore_spot_node_group_min_size = var.eks_brainstore_spot_node_group_min_size + brainstore_spot_node_group_max_size = var.eks_brainstore_spot_node_group_max_size + + enable_brainstore_writer_spot_node_group = var.eks_enable_brainstore_writer_spot_node_group + brainstore_writer_spot_node_group_instance_types = var.eks_brainstore_writer_spot_node_group_instance_types + brainstore_writer_spot_node_group_min_size = var.eks_brainstore_writer_spot_node_group_min_size + brainstore_writer_spot_node_group_max_size = var.eks_brainstore_writer_spot_node_group_max_size + + vpc_cni_addon_version = var.eks_vpc_cni_addon_version + kube_proxy_addon_version = var.eks_kube_proxy_addon_version + coredns_addon_version = var.eks_coredns_addon_version + pod_identity_addon_version = var.eks_pod_identity_addon_version + + enable_cloudfront_nlb_ingress = var.eks_enable_cloudfront_nlb_ingress + cloudfront_price_class = var.cloudfront_price_class + custom_domain = var.custom_domain + custom_certificate_arn = var.custom_certificate_arn + waf_acl_id = var.waf_acl_id + use_global_ai_proxy = var.use_global_ai_proxy + + braintrust_namespace = local.eks_namespace_resolved + braintrust_api_service_account = var.eks_api_service_account_name + brainstore_service_account = var.eks_brainstore_service_account_name + braintrust_api_role_arn = module.services_common.api_handler_role_arn + brainstore_role_arn = module.services_common.brainstore_iam_role_arn + + aws_load_balancer_controller_service_account = var.eks_aws_load_balancer_controller_service_account + + custom_tags = var.custom_tags +} diff --git a/modules/eks-cluster/README.md b/modules/eks-cluster/README.md new file mode 100644 index 00000000..e5a214af --- /dev/null +++ b/modules/eks-cluster/README.md @@ -0,0 +1,130 @@ +# EKS Cluster Module + +This module creates an Amazon EKS cluster with managed node groups for running Braintrust workloads on Kubernetes. + +## Features + +- **EKS Cluster**: Creates a fully configured EKS cluster with encryption, logging, and OIDC identity provider +- **Three Node Groups**: + - **System**: For Kubernetes system components (CoreDNS, kube-proxy, etc.) + - **Services**: For Braintrust API services + - **Brainstore**: For Brainstore analytics workloads +- **EKS Addons**: Automatically installs and manages: + - VPC CNI for networking + - CoreDNS for service discovery + - kube-proxy for service routing + - EBS CSI driver for persistent volumes + - Pod Identity agent for workload identity +- **Security**: KMS encryption for secrets, IAM roles with least privilege, security groups +- **Pod Identity Associations**: Associates the Braintrust API, Brainstore, and AWS Load Balancer Controller service accounts with IAM roles +- **Access Entries**: Optionally creates EKS access entries and access policy associations for human or CI operators +- **Auto-scaling**: All node groups support auto-scaling + +## Architecture + +Similar to the Azure AKS and GCP GKE modules, this creates: +- A managed EKS control plane +- Multiple node groups for different workload types +- Proper IAM roles and policies for cluster and nodes +- Integration with AWS services (KMS, CloudWatch) + +## Usage + +```hcl +module "eks_cluster" { + source = "./modules/eks-cluster" + + deployment_name = var.deployment_name + kubernetes_version = "1.31" + + # Networking + subnet_ids = [ + local.main_vpc_private_subnet_1_id, + local.main_vpc_private_subnet_2_id, + local.main_vpc_private_subnet_3_id + ] + + enable_private_access = true + enable_public_access = true + public_access_cidrs = ["203.0.113.10/32"] + + eks_access_entries = { + support_viewer = { + principal_arn = "arn:aws:iam::123456789012:role/BraintrustSupportViewer" + policy_associations = { + braintrust_logs = { + policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy" + access_scope = { + type = "namespace" + namespaces = ["braintrust"] + } + } + } + } + } + + # Encryption + kms_key_arn = local.kms_key_arn + + # Workload identity + braintrust_namespace = "braintrust" + braintrust_api_service_account = "braintrust-api" + brainstore_service_account = "brainstore" + braintrust_api_role_arn = module.services_common.api_handler_role_arn + brainstore_role_arn = module.services_common.brainstore_iam_role_arn + + # Node Groups + system_node_group_instance_type = "t3.medium" + system_node_group_desired_size = 2 + system_node_group_min_size = 2 + system_node_group_max_size = 4 + + services_node_group_instance_type = "r6i.2xlarge" + services_node_group_desired_size = 2 + services_node_group_min_size = 2 + services_node_group_max_size = 10 + + brainstore_node_group_instance_type = "c8gd.8xlarge" + brainstore_node_group_desired_size = 5 + brainstore_node_group_min_size = 3 + brainstore_node_group_max_size = 10 + + custom_tags = var.custom_tags +} +``` + +## Node Group Sizing + +### System Node Group +- **Default**: t3.medium +- **Purpose**: Runs Kubernetes system components +- **Sizing**: Small instances are sufficient, typically 2-4 nodes + +### Services Node Group +- **Default**: r6i.2xlarge (memory-optimized) +- **Purpose**: Runs Braintrust API services +- **Sizing**: Memory-optimized instances recommended, scale based on API load + +### Brainstore Node Group +- **Default**: c8gd.8xlarge (Graviton4, ARM64, local NVMe required) +- **AMI**: AL2023_ARM_64_STANDARD (set via `brainstore_node_group_ami_type`) +- **Purpose**: Runs Brainstore analytics workloads +- **Sizing**: Local NVMe is required for caching performance. The default c8gd.8xlarge (32 vCPU / 64 GiB) fits the brainstore writer pod and leaves capacity for readers on separate nodes. + +## Outputs + +- `cluster_id`: EKS cluster name +- `cluster_arn`: EKS cluster ARN +- `cluster_endpoint`: API server endpoint +- `cluster_oidc_issuer_url`: OIDC issuer URL for IRSA +- `cluster_certificate_authority_data`: CA certificate for kubectl +- `braintrust_api_pod_identity_association_id`: Pod Identity association for the API service account +- `brainstore_pod_identity_association_id`: Pod Identity association for the Brainstore service account +- Node group IDs and statuses + +## Requirements + +- Terraform >= 1.3 +- AWS Provider >= 5.0 +- At least 2 subnets in different availability zones +- KMS key for encryption diff --git a/modules/eks-cluster/access.tf b/modules/eks-cluster/access.tf new file mode 100644 index 00000000..174705f6 --- /dev/null +++ b/modules/eks-cluster/access.tf @@ -0,0 +1,48 @@ +#---------------------------------------------------------------------------------------------- +# EKS Access Entries +#---------------------------------------------------------------------------------------------- +locals { + eks_access_policy_associations = { + for association in flatten([ + for entry_name, entry in var.eks_access_entries : [ + for association_name, policy_association in entry.policy_associations : { + key = "${entry_name}-${association_name}" + entry_name = entry_name + policy_arn = policy_association.policy_arn + access_scope = policy_association.access_scope + } + ] + ]) : association.key => association + } +} + +resource "aws_eks_access_entry" "additional" { + for_each = var.eks_access_entries + + cluster_name = aws_eks_cluster.main.name + principal_arn = each.value.principal_arn + type = each.value.type + kubernetes_groups = each.value.kubernetes_groups + user_name = each.value.user_name + + tags = merge( + { + Name = "${var.deployment_name}-${each.key}" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_eks_access_policy_association" "additional" { + for_each = local.eks_access_policy_associations + + cluster_name = aws_eks_cluster.main.name + principal_arn = aws_eks_access_entry.additional[each.value.entry_name].principal_arn + policy_arn = each.value.policy_arn + + access_scope { + type = each.value.access_scope.type + namespaces = each.value.access_scope.type == "namespace" ? each.value.access_scope.namespaces : null + } +} diff --git a/modules/eks-cluster/assets/aws-lb-controller-iam-policy.json b/modules/eks-cluster/assets/aws-lb-controller-iam-policy.json new file mode 100644 index 00000000..a15ae556 --- /dev/null +++ b/modules/eks-cluster/assets/aws-lb-controller-iam-policy.json @@ -0,0 +1,250 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "iam:CreateServiceLinkedRole" + ], + "Resource": "*", + "Condition": { + "StringEquals": { + "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com" + } + } + }, + { + "Effect": "Allow", + "Action": [ + "ec2:DescribeAccountAttributes", + "ec2:DescribeAddresses", + "ec2:DescribeAvailabilityZones", + "ec2:DescribeInternetGateways", + "ec2:DescribeVpcs", + "ec2:DescribeVpcPeeringConnections", + "ec2:DescribeSubnets", + "ec2:DescribeSecurityGroups", + "ec2:DescribeInstances", + "ec2:DescribeNetworkInterfaces", + "ec2:DescribeTags", + "ec2:GetCoipPoolUsage", + "ec2:DescribeCoipPools", + "ec2:GetSecurityGroupsForVpc", + "ec2:DescribeIpamPools", + "elasticloadbalancing:DescribeLoadBalancers", + "elasticloadbalancing:DescribeLoadBalancerAttributes", + "elasticloadbalancing:DescribeListeners", + "elasticloadbalancing:DescribeListenerCertificates", + "elasticloadbalancing:DescribeSSLPolicies", + "elasticloadbalancing:DescribeRules", + "elasticloadbalancing:DescribeTargetGroups", + "elasticloadbalancing:DescribeTargetGroupAttributes", + "elasticloadbalancing:DescribeTargetHealth", + "elasticloadbalancing:DescribeTags", + "elasticloadbalancing:DescribeTrustStores", + "elasticloadbalancing:DescribeListenerAttributes", + "elasticloadbalancing:DescribeCapacityReservation" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "cognito-idp:DescribeUserPoolClient", + "acm:ListCertificates", + "acm:DescribeCertificate", + "iam:ListServerCertificates", + "iam:GetServerCertificate", + "waf-regional:GetWebACL", + "waf-regional:GetWebACLForResource", + "waf-regional:AssociateWebACL", + "waf-regional:DisassociateWebACL", + "wafv2:GetWebACL", + "wafv2:GetWebACLForResource", + "wafv2:AssociateWebACL", + "wafv2:DisassociateWebACL", + "shield:GetSubscriptionState", + "shield:DescribeProtection", + "shield:CreateProtection", + "shield:DeleteProtection" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "ec2:AuthorizeSecurityGroupIngress", + "ec2:RevokeSecurityGroupIngress" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "ec2:CreateSecurityGroup" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "ec2:CreateTags" + ], + "Resource": "arn:aws:ec2:*:*:security-group/*", + "Condition": { + "StringEquals": { + "ec2:CreateAction": "CreateSecurityGroup" + }, + "Null": { + "aws:RequestTag/elbv2.k8s.aws/cluster": "false" + } + } + }, + { + "Effect": "Allow", + "Action": [ + "ec2:CreateTags", + "ec2:DeleteTags" + ], + "Resource": "arn:aws:ec2:*:*:security-group/*", + "Condition": { + "Null": { + "aws:RequestTag/elbv2.k8s.aws/cluster": "true", + "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" + } + } + }, + { + "Effect": "Allow", + "Action": [ + "ec2:AuthorizeSecurityGroupIngress", + "ec2:RevokeSecurityGroupIngress", + "ec2:DeleteSecurityGroup" + ], + "Resource": "*", + "Condition": { + "Null": { + "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" + } + } + }, + { + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:CreateLoadBalancer", + "elasticloadbalancing:CreateTargetGroup" + ], + "Resource": "*", + "Condition": { + "Null": { + "aws:RequestTag/elbv2.k8s.aws/cluster": "false" + } + } + }, + { + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:CreateListener", + "elasticloadbalancing:DeleteListener", + "elasticloadbalancing:CreateRule", + "elasticloadbalancing:DeleteRule" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:AddTags", + "elasticloadbalancing:RemoveTags" + ], + "Resource": [ + "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*", + "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*", + "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*" + ], + "Condition": { + "Null": { + "aws:RequestTag/elbv2.k8s.aws/cluster": "true", + "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" + } + } + }, + { + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:AddTags", + "elasticloadbalancing:RemoveTags" + ], + "Resource": [ + "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*", + "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*", + "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*", + "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:ModifyLoadBalancerAttributes", + "elasticloadbalancing:SetIpAddressType", + "elasticloadbalancing:SetSecurityGroups", + "elasticloadbalancing:SetSubnets", + "elasticloadbalancing:DeleteLoadBalancer", + "elasticloadbalancing:ModifyTargetGroup", + "elasticloadbalancing:ModifyTargetGroupAttributes", + "elasticloadbalancing:DeleteTargetGroup", + "elasticloadbalancing:ModifyListenerAttributes", + "elasticloadbalancing:ModifyCapacityReservation", + "elasticloadbalancing:ModifyIpPools" + ], + "Resource": "*", + "Condition": { + "Null": { + "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" + } + } + }, + { + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:AddTags" + ], + "Resource": [ + "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*", + "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*", + "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*" + ], + "Condition": { + "StringEquals": { + "elasticloadbalancing:CreateAction": [ + "CreateTargetGroup", + "CreateLoadBalancer" + ] + }, + "Null": { + "aws:RequestTag/elbv2.k8s.aws/cluster": "false" + } + } + }, + { + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:RegisterTargets", + "elasticloadbalancing:DeregisterTargets" + ], + "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*" + }, + { + "Effect": "Allow", + "Action": [ + "elasticloadbalancing:SetWebAcl", + "elasticloadbalancing:ModifyListener", + "elasticloadbalancing:AddListenerCertificates", + "elasticloadbalancing:RemoveListenerCertificates", + "elasticloadbalancing:ModifyRule", + "elasticloadbalancing:SetRulePriorities" + ], + "Resource": "*" + } + ] +} diff --git a/modules/eks-cluster/cloudfront.tf b/modules/eks-cluster/cloudfront.tf new file mode 100644 index 00000000..148bc87c --- /dev/null +++ b/modules/eks-cluster/cloudfront.tf @@ -0,0 +1,128 @@ +locals { + cloudfront_CachingDisabled = "4135ea2d-6df8-44a3-9df3-4b5a84be39ad" + cloudfront_AllViewerExceptHostHeader = "b689b0a8-53d0-40ab-baf2-68738e2966ac" +} + +resource "aws_cloudfront_vpc_origin" "api" { + count = var.enable_cloudfront_nlb_ingress ? 1 : 0 + + vpc_origin_endpoint_config { + name = "${var.deployment_name}-eks-api" + arn = aws_lb.api[0].arn + http_port = 8000 + https_port = 443 + origin_protocol_policy = "http-only" + + origin_ssl_protocols { + items = ["TLSv1.2"] + quantity = 1 + } + } + + tags = merge( + { + Name = "${var.deployment_name}-eks-api" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_cloudfront_distribution" "dataplane" { + count = var.enable_cloudfront_nlb_ingress ? 1 : 0 + comment = "Braintrust EKS Dataplane - ${var.deployment_name}" + enabled = true + http_version = "http2and3" + web_acl_id = var.waf_acl_id + price_class = var.cloudfront_price_class + aliases = var.custom_domain != null ? [var.custom_domain] : null + + origin { + origin_id = "EKSAPIOrigin" + domain_name = aws_lb.api[0].dns_name + + vpc_origin_config { + vpc_origin_id = aws_cloudfront_vpc_origin.api[0].id + origin_read_timeout = 60 + origin_keepalive_timeout = 60 + } + + dynamic "custom_header" { + for_each = var.custom_domain != null ? [1] : [] + content { + name = "X-CloudFront-Domain" + value = var.custom_domain + } + } + } + + origin { + domain_name = "braintrustproxy.com" + origin_id = "CloudflareProxy" + + custom_origin_config { + origin_protocol_policy = "https-only" + origin_read_timeout = 60 + origin_keepalive_timeout = 60 + https_port = 443 + http_port = 80 + origin_ssl_protocols = ["TLSv1.2"] + } + } + + default_cache_behavior { + allowed_methods = ["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"] + cached_methods = ["GET", "HEAD", "OPTIONS"] + target_origin_id = "EKSAPIOrigin" + viewer_protocol_policy = "redirect-to-https" + compress = true + + cache_policy_id = local.cloudfront_CachingDisabled + origin_request_policy_id = local.cloudfront_AllViewerExceptHostHeader + } + + dynamic "ordered_cache_behavior" { + for_each = toset([ + "/v1/proxy", + "/v1/proxy/*", + "/v1/eval", + "/v1/eval/*", + "/v1/function/*/?*", + "/function/*", + ]) + content { + path_pattern = ordered_cache_behavior.value + allowed_methods = ["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"] + cached_methods = ["GET", "HEAD", "OPTIONS"] + target_origin_id = var.use_global_ai_proxy ? "CloudflareProxy" : "EKSAPIOrigin" + viewer_protocol_policy = "redirect-to-https" + compress = true + + cache_policy_id = local.cloudfront_CachingDisabled + origin_request_policy_id = local.cloudfront_AllViewerExceptHostHeader + } + } + + viewer_certificate { + cloudfront_default_certificate = var.custom_certificate_arn != null ? false : true + acm_certificate_arn = var.custom_certificate_arn + minimum_protocol_version = var.custom_certificate_arn != null ? "TLSv1.3_2025" : null + ssl_support_method = var.custom_certificate_arn != null ? "sni-only" : null + } + + restrictions { + geo_restriction { + restriction_type = "none" + } + } + + depends_on = [aws_cloudfront_vpc_origin.api] + + tags = merge( + { + Name = "${var.deployment_name}-eks-dataplane" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} diff --git a/modules/eks-cluster/iam.tf b/modules/eks-cluster/iam.tf new file mode 100644 index 00000000..decdf0f4 --- /dev/null +++ b/modules/eks-cluster/iam.tf @@ -0,0 +1,159 @@ +# IAM roles used by Kubernetes control-plane add-ons. Workload roles for the +# Braintrust API and Brainstore are created by services-common and associated in +# this module. + +#---------------------------------------------------------------------------------------------- +# EKS Auto Mode IAM — Cluster Role +# These four managed policies are required for the cluster role when EKS Auto Mode is enabled. +# They grant EKS permission to manage compute (node pools), block storage (EBS), load balancing +# (NLBs/ALBs), and networking (ENIs/IPs) on your behalf. +#---------------------------------------------------------------------------------------------- +resource "aws_iam_role_policy_attachment" "cluster_compute_policy" { + count = var.use_eks_auto_mode ? 1 : 0 + policy_arn = "arn:aws:iam::aws:policy/AmazonEKSComputePolicy" + role = aws_iam_role.cluster.name +} + +resource "aws_iam_role_policy_attachment" "cluster_block_storage_policy" { + count = var.use_eks_auto_mode ? 1 : 0 + policy_arn = "arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy" + role = aws_iam_role.cluster.name +} + +resource "aws_iam_role_policy_attachment" "cluster_load_balancing_policy" { + count = var.use_eks_auto_mode ? 1 : 0 + policy_arn = "arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy" + role = aws_iam_role.cluster.name +} + +resource "aws_iam_role_policy_attachment" "cluster_networking_policy" { + count = var.use_eks_auto_mode ? 1 : 0 + policy_arn = "arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy" + role = aws_iam_role.cluster.name +} + +#---------------------------------------------------------------------------------------------- +# EKS Auto Mode IAM — Node Role +# Auto Mode-provisioned nodes use a reduced permission set compared to traditional managed +# node groups. AmazonEKSWorkerNodeMinimalPolicy replaces AmazonEKSWorkerNodePolicy, and +# AmazonEC2ContainerRegistryPullOnly (read-only) replaces the broader ReadOnly policy. +# These are attached alongside the standard policies so the same role works for both +# Auto Mode node pools and the Brainstore managed node groups. +#---------------------------------------------------------------------------------------------- +resource "aws_iam_role_policy_attachment" "node_group_worker_minimal_policy" { + count = var.use_eks_auto_mode ? 1 : 0 + policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy" + role = aws_iam_role.node_group.name +} + +resource "aws_iam_role_policy_attachment" "node_group_ecr_pull_only_policy" { + count = var.use_eks_auto_mode ? 1 : 0 + policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly" + role = aws_iam_role.node_group.name +} + +resource "aws_iam_policy" "lb_controller" { + name = "${var.deployment_name}-lb-controller" + policy = file("${path.module}/assets/aws-lb-controller-iam-policy.json") + + tags = merge( + { + Name = "${var.deployment_name}-lb-controller" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_iam_role" "lb_controller" { + name = "${var.deployment_name}-lb-controller" + + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Effect = "Allow" + Principal = { + Service = "pods.eks.amazonaws.com" + } + Action = [ + "sts:AssumeRole", + "sts:TagSession" + ] + } + ] + }) + + permissions_boundary = var.permissions_boundary_arn + + tags = merge( + { + Name = "${var.deployment_name}-lb-controller" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_iam_role_policy_attachment" "lb_controller" { + role = aws_iam_role.lb_controller.name + policy_arn = aws_iam_policy.lb_controller.arn +} + +resource "aws_eks_pod_identity_association" "lb_controller" { + cluster_name = aws_eks_cluster.main.name + namespace = "kube-system" + service_account = var.aws_load_balancer_controller_service_account + role_arn = aws_iam_role.lb_controller.arn + + tags = merge( + { + Name = "${var.deployment_name}-lb-controller" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + depends_on = [ + aws_eks_addon.pod_identity, + aws_iam_role_policy_attachment.lb_controller, + ] +} + +resource "aws_eks_pod_identity_association" "braintrust_api" { + cluster_name = aws_eks_cluster.main.name + namespace = var.braintrust_namespace + service_account = var.braintrust_api_service_account + role_arn = var.braintrust_api_role_arn + + tags = merge( + { + Name = "${var.deployment_name}-braintrust-api" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + depends_on = [ + aws_eks_addon.pod_identity, + ] +} + +resource "aws_eks_pod_identity_association" "brainstore" { + cluster_name = aws_eks_cluster.main.name + namespace = var.braintrust_namespace + service_account = var.brainstore_service_account + role_arn = var.brainstore_role_arn + + tags = merge( + { + Name = "${var.deployment_name}-brainstore" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + depends_on = [ + aws_eks_addon.pod_identity, + ] +} diff --git a/modules/eks-cluster/main.tf b/modules/eks-cluster/main.tf new file mode 100644 index 00000000..4b0f27a7 --- /dev/null +++ b/modules/eks-cluster/main.tf @@ -0,0 +1,764 @@ +#---------------------------------------------------------------------------------------------- +# EKS Cluster IAM Role +#---------------------------------------------------------------------------------------------- +resource "aws_iam_role" "cluster" { + name = "${var.deployment_name}-eks-cluster" + + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Action = [ + "sts:AssumeRole", + "sts:TagSession" + ] + Effect = "Allow" + Principal = { + Service = "eks.amazonaws.com" + } + } + ] + }) + + permissions_boundary = var.permissions_boundary_arn + + tags = merge( + { + Name = "${var.deployment_name}-eks-cluster" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_iam_role_policy_attachment" "cluster_policy" { + policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy" + role = aws_iam_role.cluster.name +} + +#---------------------------------------------------------------------------------------------- +# CloudWatch Log Group for EKS Control Plane +# Created before the cluster so Terraform controls retention and encryption. +# EKS auto-creates this log group if it doesn't exist — but without retention or tags. +#---------------------------------------------------------------------------------------------- +resource "aws_cloudwatch_log_group" "cluster" { + name = "/aws/eks/${var.deployment_name}-eks/cluster" + retention_in_days = var.cluster_log_retention_days == 0 ? null : var.cluster_log_retention_days + + tags = merge( + { + Name = "${var.deployment_name}-eks-logs" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +#---------------------------------------------------------------------------------------------- +# EKS Cluster +#---------------------------------------------------------------------------------------------- +resource "aws_eks_cluster" "main" { + name = "${var.deployment_name}-eks" + role_arn = aws_iam_role.cluster.arn + version = var.kubernetes_version + + access_config { + authentication_mode = "API" + bootstrap_cluster_creator_admin_permissions = true + } + + vpc_config { + subnet_ids = var.subnet_ids + + endpoint_private_access = var.enable_private_access + endpoint_public_access = var.enable_public_access + public_access_cidrs = var.public_access_cidrs + + security_group_ids = var.additional_security_group_ids + } + + encryption_config { + provider { + key_arn = var.kms_key_arn + } + resources = ["secrets"] + } + + enabled_cluster_log_types = var.cluster_log_types + + # Required by EKS API when Auto Mode (compute_config.enabled = true) is used. + bootstrap_self_managed_addons = var.use_eks_auto_mode ? false : null + + # EKS Auto Mode: compute, load balancing, and block storage managed by EKS. + # System and services node pools are handled by built-in node pools ("system", "general-purpose"). + # Brainstore nodes still use managed node groups for NVMe launch template support. + dynamic "compute_config" { + for_each = var.use_eks_auto_mode ? [1] : [] + content { + enabled = true + node_pools = ["general-purpose", "system"] + node_role_arn = aws_iam_role.node_group.arn + } + } + + dynamic "kubernetes_network_config" { + for_each = var.use_eks_auto_mode ? [1] : [] + content { + elastic_load_balancing { + enabled = true + } + } + } + + dynamic "storage_config" { + for_each = var.use_eks_auto_mode ? [1] : [] + content { + block_storage { + enabled = true + } + } + } + + tags = merge( + { + Name = "${var.deployment_name}-eks" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + depends_on = [ + aws_iam_role_policy_attachment.cluster_policy, + aws_cloudwatch_log_group.cluster, + aws_iam_role_policy_attachment.cluster_compute_policy, + aws_iam_role_policy_attachment.cluster_block_storage_policy, + aws_iam_role_policy_attachment.cluster_load_balancing_policy, + aws_iam_role_policy_attachment.cluster_networking_policy, + aws_iam_role_policy_attachment.node_group_worker_minimal_policy, + aws_iam_role_policy_attachment.node_group_ecr_pull_only_policy, + ] +} + +#---------------------------------------------------------------------------------------------- +# EKS Node Group IAM Role +#---------------------------------------------------------------------------------------------- +resource "aws_iam_role" "node_group" { + name = "${var.deployment_name}-eks-node-group" + + assume_role_policy = jsonencode({ + Statement = [{ + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "ec2.amazonaws.com" + } + }] + Version = "2012-10-17" + }) + + permissions_boundary = var.permissions_boundary_arn + + tags = merge( + { + Name = "${var.deployment_name}-eks-node-group" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_iam_role_policy_attachment" "node_group_worker_policy" { + policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy" + role = aws_iam_role.node_group.name +} + +resource "aws_iam_role_policy_attachment" "node_group_cni_policy" { + policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy" + role = aws_iam_role.node_group.name +} + +resource "aws_iam_role_policy_attachment" "node_group_ecr_policy" { + policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" + role = aws_iam_role.node_group.name +} + +# Optional: SSM access for node debugging +resource "aws_iam_role_policy_attachment" "node_group_ssm_policy" { + count = var.enable_node_ssm ? 1 : 0 + policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" + role = aws_iam_role.node_group.name +} + +#---------------------------------------------------------------------------------------------- +# System Node Group (for system pods like CoreDNS, kube-proxy, etc.) +# Not created when use_eks_auto_mode = true — EKS Auto Mode's built-in "system" node pool +# handles kube-system workloads automatically. +#---------------------------------------------------------------------------------------------- +resource "aws_eks_node_group" "system" { + count = var.use_eks_auto_mode ? 0 : 1 + + cluster_name = aws_eks_cluster.main.name + node_group_name = "${var.deployment_name}-system" + node_role_arn = aws_iam_role.node_group.arn + subnet_ids = var.subnet_ids + version = var.kubernetes_version + ami_type = var.node_group_ami_type + + scaling_config { + desired_size = var.system_node_group_desired_size + max_size = var.system_node_group_max_size + min_size = var.system_node_group_min_size + } + + update_config { + max_unavailable = 1 + } + + disk_size = var.system_node_group_disk_size + instance_types = [var.system_node_group_instance_type] + capacity_type = "ON_DEMAND" + + labels = { + role = "system" + } + + tags = merge( + { + Name = "${var.deployment_name}-system" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + lifecycle { + ignore_changes = [scaling_config[0].desired_size] + } + + depends_on = [ + aws_iam_role_policy_attachment.node_group_worker_policy, + aws_iam_role_policy_attachment.node_group_cni_policy, + aws_iam_role_policy_attachment.node_group_ecr_policy + ] +} + +#---------------------------------------------------------------------------------------------- +# Services Node Group (for Braintrust API services) +# Not created when use_eks_auto_mode = true — EKS Auto Mode's built-in "general-purpose" node +# pool handles API service workloads and scales them automatically. +#---------------------------------------------------------------------------------------------- +resource "aws_eks_node_group" "services" { + count = var.use_eks_auto_mode ? 0 : 1 + + cluster_name = aws_eks_cluster.main.name + node_group_name = "${var.deployment_name}-services" + node_role_arn = aws_iam_role.node_group.arn + subnet_ids = var.subnet_ids + version = var.kubernetes_version + ami_type = var.node_group_ami_type + + scaling_config { + desired_size = var.services_node_group_desired_size + max_size = var.services_node_group_max_size + min_size = var.services_node_group_min_size + } + + update_config { + max_unavailable = 1 + } + + disk_size = var.services_node_group_disk_size + instance_types = [var.services_node_group_instance_type] + capacity_type = "ON_DEMAND" + + labels = { + role = "services" + } + + # Prevent unintended workloads from landing on services nodes. + # API pods must include a matching toleration in their Helm values. + taint { + key = "dedicated" + value = "services" + effect = "NO_SCHEDULE" + } + + tags = merge( + { + Name = "${var.deployment_name}-services" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + lifecycle { + ignore_changes = [scaling_config[0].desired_size] + } + + depends_on = [ + aws_iam_role_policy_attachment.node_group_worker_policy, + aws_iam_role_policy_attachment.node_group_cni_policy, + aws_iam_role_policy_attachment.node_group_ecr_policy + ] +} + +#---------------------------------------------------------------------------------------------- +# Services Spot Node Group (burst capacity for Braintrust API services) +# Creates an optional fixed-capacity SPOT pool alongside the on-demand baseline. +# Not created when use_eks_auto_mode = true — Auto Mode handles burst scaling natively. +#---------------------------------------------------------------------------------------------- +resource "aws_eks_node_group" "services_spot" { + count = (!var.use_eks_auto_mode && var.enable_services_spot_node_group) ? 1 : 0 + + cluster_name = aws_eks_cluster.main.name + node_group_name = "${var.deployment_name}-services-spot" + node_role_arn = aws_iam_role.node_group.arn + subnet_ids = var.subnet_ids + version = var.kubernetes_version + ami_type = var.node_group_ami_type + + scaling_config { + desired_size = var.services_spot_node_group_min_size + max_size = var.services_spot_node_group_max_size + min_size = var.services_spot_node_group_min_size + } + + update_config { + max_unavailable = 1 + } + + disk_size = var.services_node_group_disk_size + instance_types = var.services_spot_node_group_instance_types + capacity_type = "SPOT" + + labels = { + role = "services" + } + + # Must match the taint on the on-demand services node group so that API pods + # with a services toleration can land on either node group. + taint { + key = "dedicated" + value = "services" + effect = "NO_SCHEDULE" + } + + tags = merge( + { + Name = "${var.deployment_name}-services-spot" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + lifecycle { + ignore_changes = [scaling_config[0].desired_size] + } + + depends_on = [ + aws_iam_role_policy_attachment.node_group_worker_policy, + aws_iam_role_policy_attachment.node_group_cni_policy, + aws_iam_role_policy_attachment.node_group_ecr_policy + ] +} + +#---------------------------------------------------------------------------------------------- +# Brainstore Reader Node Group (for reader and fast-reader pods) +#---------------------------------------------------------------------------------------------- +resource "aws_eks_node_group" "brainstore_reader" { + count = var.use_eks_auto_mode ? 0 : 1 + cluster_name = aws_eks_cluster.main.name + node_group_name = "${var.deployment_name}-brainstore-reader" + node_role_arn = aws_iam_role.node_group.arn + subnet_ids = var.subnet_ids + version = var.kubernetes_version + ami_type = var.brainstore_node_group_ami_type + + scaling_config { + desired_size = var.brainstore_reader_node_group_desired_size + max_size = var.brainstore_reader_node_group_max_size + min_size = var.brainstore_reader_node_group_min_size + } + + update_config { + max_unavailable = 1 + } + + instance_types = [var.brainstore_reader_node_group_instance_type] + capacity_type = "ON_DEMAND" + + labels = { + role = "brainstore-reader" + } + + # Prevent unintended workloads from landing on brainstore reader nodes. + # Reader and fast-reader pods must include a matching toleration in their Helm values. + taint { + key = "dedicated" + value = "brainstore" + effect = "NO_SCHEDULE" + } + + # Launch template formats and mounts the local NVMe instance store before node joins the cluster + launch_template { + id = aws_launch_template.brainstore_nvme[0].id + version = tostring(aws_launch_template.brainstore_nvme[0].latest_version) + } + + tags = merge( + { + Name = "${var.deployment_name}-brainstore-reader" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + lifecycle { + ignore_changes = [scaling_config[0].desired_size] + } + + depends_on = [ + aws_iam_role_policy_attachment.node_group_worker_policy, + aws_iam_role_policy_attachment.node_group_cni_policy, + aws_iam_role_policy_attachment.node_group_ecr_policy + ] +} + +#---------------------------------------------------------------------------------------------- +# Brainstore Writer Node Group (for writer pods — 2× CPU/memory of the reader nodes) +#---------------------------------------------------------------------------------------------- +resource "aws_eks_node_group" "brainstore_writer" { + count = var.use_eks_auto_mode ? 0 : 1 + cluster_name = aws_eks_cluster.main.name + node_group_name = "${var.deployment_name}-brainstore-writer" + node_role_arn = aws_iam_role.node_group.arn + subnet_ids = var.subnet_ids + version = var.kubernetes_version + ami_type = var.brainstore_node_group_ami_type + + scaling_config { + desired_size = var.brainstore_writer_node_group_desired_size + max_size = var.brainstore_writer_node_group_max_size + min_size = var.brainstore_writer_node_group_min_size + } + + update_config { + max_unavailable = 1 + } + + instance_types = [var.brainstore_writer_node_group_instance_type] + capacity_type = "ON_DEMAND" + + labels = { + role = "brainstore-writer" + } + + # Prevent unintended workloads from landing on brainstore writer nodes. + # Writer pods must include a matching toleration in their Helm values. + taint { + key = "dedicated" + value = "brainstore" + effect = "NO_SCHEDULE" + } + + # Launch template formats and mounts the local NVMe instance store before node joins the cluster + launch_template { + id = aws_launch_template.brainstore_nvme[0].id + version = tostring(aws_launch_template.brainstore_nvme[0].latest_version) + } + + tags = merge( + { + Name = "${var.deployment_name}-brainstore-writer" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + lifecycle { + ignore_changes = [scaling_config[0].desired_size] + } + + depends_on = [ + aws_iam_role_policy_attachment.node_group_worker_policy, + aws_iam_role_policy_attachment.node_group_cni_policy, + aws_iam_role_policy_attachment.node_group_ecr_policy + ] +} + +#---------------------------------------------------------------------------------------------- +# Brainstore Spot Node Group (burst capacity for Brainstore workloads) +# Creates an optional fixed-capacity SPOT pool alongside the on-demand baseline. All instance types must be +# NVMe-backed Graviton instances so the pre-bootstrap mount script runs correctly on every node. +# Spot interruptions cause cache loss; Brainstore pods rehydrate from S3 on restart. +# Set enable_brainstore_spot_node_group = true to create this group. +#---------------------------------------------------------------------------------------------- +resource "aws_eks_node_group" "brainstore_spot" { + count = !var.use_eks_auto_mode && var.enable_brainstore_spot_node_group ? 1 : 0 + + cluster_name = aws_eks_cluster.main.name + node_group_name = "${var.deployment_name}-brainstore-spot" + node_role_arn = aws_iam_role.node_group.arn + subnet_ids = var.subnet_ids + version = var.kubernetes_version + ami_type = var.brainstore_node_group_ami_type + + scaling_config { + desired_size = var.brainstore_spot_node_group_min_size + max_size = var.brainstore_spot_node_group_max_size + min_size = var.brainstore_spot_node_group_min_size + } + + update_config { + max_unavailable = 1 + } + + instance_types = var.brainstore_spot_node_group_instance_types + capacity_type = "SPOT" + + labels = { + role = "brainstore-reader" + } + + # Must match the taint on the on-demand brainstore reader node group so that reader + # and fast-reader pods can land on either node group. + taint { + key = "dedicated" + value = "brainstore" + effect = "NO_SCHEDULE" + } + + # Same NVMe launch template as the on-demand group — formats and mounts + # local NVMe instance store before the node joins the cluster. + launch_template { + id = aws_launch_template.brainstore_nvme[0].id + version = tostring(aws_launch_template.brainstore_nvme[0].latest_version) + } + + tags = merge( + { + Name = "${var.deployment_name}-brainstore-spot" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + lifecycle { + ignore_changes = [scaling_config[0].desired_size] + } + + depends_on = [ + aws_iam_role_policy_attachment.node_group_worker_policy, + aws_iam_role_policy_attachment.node_group_cni_policy, + aws_iam_role_policy_attachment.node_group_ecr_policy + ] +} + +#---------------------------------------------------------------------------------------------- +# Brainstore Writer Spot Node Group (burst capacity for writer workloads) +# Uses larger NVMe-backed instances (2× reader size) to match writer resource requirements. +# Set enable_brainstore_writer_spot_node_group = true to create this group. +#---------------------------------------------------------------------------------------------- +resource "aws_eks_node_group" "brainstore_writer_spot" { + count = !var.use_eks_auto_mode && var.enable_brainstore_writer_spot_node_group ? 1 : 0 + + cluster_name = aws_eks_cluster.main.name + node_group_name = "${var.deployment_name}-brainstore-writer-spot" + node_role_arn = aws_iam_role.node_group.arn + subnet_ids = var.subnet_ids + version = var.kubernetes_version + ami_type = var.brainstore_node_group_ami_type + + scaling_config { + desired_size = var.brainstore_writer_spot_node_group_min_size + max_size = var.brainstore_writer_spot_node_group_max_size + min_size = var.brainstore_writer_spot_node_group_min_size + } + + update_config { + max_unavailable = 1 + } + + instance_types = var.brainstore_writer_spot_node_group_instance_types + capacity_type = "SPOT" + + labels = { + role = "brainstore-writer" + } + + # Must match the taint on the on-demand brainstore writer node group so that + # writer pods can land on either node group. + taint { + key = "dedicated" + value = "brainstore" + effect = "NO_SCHEDULE" + } + + launch_template { + id = aws_launch_template.brainstore_nvme[0].id + version = tostring(aws_launch_template.brainstore_nvme[0].latest_version) + } + + tags = merge( + { + Name = "${var.deployment_name}-brainstore-writer-spot" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + lifecycle { + ignore_changes = [scaling_config[0].desired_size] + } + + depends_on = [ + aws_iam_role_policy_attachment.node_group_worker_policy, + aws_iam_role_policy_attachment.node_group_cni_policy, + aws_iam_role_policy_attachment.node_group_ecr_policy + ] +} + +#---------------------------------------------------------------------------------------------- +# Brainstore NVMe Launch Template +# Formats and mounts local NVMe instance store volumes before the node joins the cluster. +# Runs as a pre-bootstrap shell script via MIME multipart user data. +# +# The script auto-detects all NVMe instance store devices at boot: +# 1 device — formatted and mounted directly +# 2+ devices — striped as RAID 0 for maximum aggregate throughput +# +# Tested with c8gd (Graviton4 NVMe) instances. Also compatible with c6id and other +# NVMe-backed families. +#---------------------------------------------------------------------------------------------- +resource "aws_launch_template" "brainstore_nvme" { + count = var.use_eks_auto_mode ? 0 : 1 + name_prefix = "${var.deployment_name}-brainstore-" + description = "Formats and mounts local NVMe instance store for Brainstore nodes" + + # MIME multipart user data is required for AL2023 managed node groups. + # The shell script runs before EKS bootstraps the node, ensuring the NVMe + # volume is mounted before any pods are scheduled. + user_data = base64encode(<<-EOT + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="==BOUNDARY==" + + --==BOUNDARY== + Content-Type: application/node.eks.aws + + apiVersion: node.eks.aws/v1alpha1 + kind: NodeConfig + spec: + instance: + localStorage: + strategy: RAID0 + + --==BOUNDARY==-- + EOT + ) + + tag_specifications { + resource_type = "instance" + tags = merge( + { + Name = "${var.deployment_name}-brainstore" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + } + + tags = merge( + { + Name = "${var.deployment_name}-brainstore-nvme-lt" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + lifecycle { + create_before_destroy = true + } +} + +#---------------------------------------------------------------------------------------------- +# EKS Addons +# vpc-cni, kube-proxy, and coredns are managed by EKS Auto Mode when use_eks_auto_mode = true. +# eks-pod-identity-agent is always installed regardless of mode. +#---------------------------------------------------------------------------------------------- +resource "aws_eks_addon" "vpc_cni" { + count = var.use_eks_auto_mode ? 0 : 1 + + cluster_name = aws_eks_cluster.main.name + addon_name = "vpc-cni" + addon_version = var.vpc_cni_addon_version != null ? var.vpc_cni_addon_version : null + + resolve_conflicts_on_create = "OVERWRITE" + resolve_conflicts_on_update = "OVERWRITE" + + tags = merge( + { + Name = "${var.deployment_name}-vpc-cni" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_eks_addon" "kube_proxy" { + count = var.use_eks_auto_mode ? 0 : 1 + + cluster_name = aws_eks_cluster.main.name + addon_name = "kube-proxy" + addon_version = var.kube_proxy_addon_version != null ? var.kube_proxy_addon_version : null + + resolve_conflicts_on_create = "OVERWRITE" + resolve_conflicts_on_update = "OVERWRITE" + + tags = merge( + { + Name = "${var.deployment_name}-kube-proxy" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_eks_addon" "coredns" { + count = var.use_eks_auto_mode ? 0 : 1 + + cluster_name = aws_eks_cluster.main.name + addon_name = "coredns" + addon_version = var.coredns_addon_version != null ? var.coredns_addon_version : null + + resolve_conflicts_on_create = "OVERWRITE" + resolve_conflicts_on_update = "OVERWRITE" + + tags = merge( + { + Name = "${var.deployment_name}-coredns" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) + + depends_on = [ + aws_eks_node_group.system + ] +} + + +resource "aws_eks_addon" "pod_identity" { + cluster_name = aws_eks_cluster.main.name + addon_name = "eks-pod-identity-agent" + addon_version = var.pod_identity_addon_version != null ? var.pod_identity_addon_version : null + + resolve_conflicts_on_create = "OVERWRITE" + resolve_conflicts_on_update = "OVERWRITE" + + tags = merge( + { + Name = "${var.deployment_name}-pod-identity-agent" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} diff --git a/modules/eks-cluster/networking.tf b/modules/eks-cluster/networking.tf new file mode 100644 index 00000000..d0291325 --- /dev/null +++ b/modules/eks-cluster/networking.tf @@ -0,0 +1,77 @@ +# Tag private subnets so the AWS Load Balancer Controller can discover them for +# internal load balancers. +resource "aws_ec2_tag" "private_subnet_internal_elb" { + count = length(var.subnet_ids) + resource_id = var.subnet_ids[count.index] + key = "kubernetes.io/role/internal-elb" + value = "1" +} + +data "aws_ec2_managed_prefix_list" "cloudfront" { + count = var.enable_cloudfront_nlb_ingress ? 1 : 0 + name = "com.amazonaws.global.cloudfront.origin-facing" +} + +resource "aws_security_group" "nlb_cloudfront" { + count = var.enable_cloudfront_nlb_ingress ? 1 : 0 + name = "${var.deployment_name}-nlb-cloudfront" + description = "Allow CloudFront VPC Origin traffic to the Braintrust API NLB" + vpc_id = var.vpc_id + + tags = merge( + { + Name = "${var.deployment_name}-nlb-cloudfront" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} + +resource "aws_vpc_security_group_ingress_rule" "nlb_from_cloudfront" { + count = var.enable_cloudfront_nlb_ingress ? 1 : 0 + security_group_id = aws_security_group.nlb_cloudfront[0].id + prefix_list_id = data.aws_ec2_managed_prefix_list.cloudfront[0].id + from_port = 8000 + to_port = 8000 + ip_protocol = "tcp" + description = "CloudFront VPC Origin to Braintrust API" +} + +resource "aws_vpc_security_group_egress_rule" "nlb_all_egress" { + count = var.enable_cloudfront_nlb_ingress ? 1 : 0 + security_group_id = aws_security_group.nlb_cloudfront[0].id + cidr_ipv4 = "0.0.0.0/0" + ip_protocol = "-1" + description = "Allow NLB egress to EKS nodes" +} + +resource "aws_vpc_security_group_ingress_rule" "eks_nodes_from_nlb" { + count = var.enable_cloudfront_nlb_ingress ? 1 : 0 + security_group_id = aws_eks_cluster.main.vpc_config[0].cluster_security_group_id + referenced_security_group_id = aws_security_group.nlb_cloudfront[0].id + from_port = 30000 + to_port = 32767 + ip_protocol = "tcp" + description = "Braintrust API NLB to EKS NodePorts" +} + +# Pre-create the NLB so CloudFront VPC Origin can reference its ARN at plan +# time. The AWS Load Balancer Controller adopts this NLB through the +# service.beta.kubernetes.io/aws-load-balancer-name annotation in the Helm +# values emitted by the eks-deploy module. +resource "aws_lb" "api" { + count = var.enable_cloudfront_nlb_ingress ? 1 : 0 + name = "${var.deployment_name}-api-nlb" + internal = true + load_balancer_type = "network" + security_groups = [aws_security_group.nlb_cloudfront[0].id] + subnets = var.subnet_ids + + tags = merge( + { + Name = "${var.deployment_name}-api-nlb" + BraintrustDeploymentName = var.deployment_name + }, + var.custom_tags + ) +} diff --git a/modules/eks-cluster/outputs.tf b/modules/eks-cluster/outputs.tf new file mode 100644 index 00000000..33f1836e --- /dev/null +++ b/modules/eks-cluster/outputs.tf @@ -0,0 +1,175 @@ +output "cluster_id" { + description = "The name of the EKS cluster" + value = aws_eks_cluster.main.id +} + +output "cluster_name" { + description = "The name of the EKS cluster (alias for cluster_id)" + value = aws_eks_cluster.main.name +} + +output "cluster_arn" { + description = "The ARN of the EKS cluster" + value = aws_eks_cluster.main.arn +} + +output "cluster_endpoint" { + description = "Endpoint for the EKS cluster API server" + value = aws_eks_cluster.main.endpoint +} + +output "cluster_security_group_id" { + description = "Security group ID attached to the EKS cluster" + value = aws_eks_cluster.main.vpc_config[0].cluster_security_group_id +} + +output "cluster_certificate_authority_data" { + description = "Base64 encoded certificate data for cluster authentication" + value = aws_eks_cluster.main.certificate_authority[0].data + sensitive = true +} + +output "cluster_oidc_issuer_url" { + description = "The URL on the EKS cluster OIDC issuer (for IRSA)" + value = try(aws_eks_cluster.main.identity[0].oidc[0].issuer, "") +} + +output "cluster_platform_version" { + description = "Platform version for the EKS cluster" + value = aws_eks_cluster.main.platform_version +} + +output "cluster_status" { + description = "Status of the EKS cluster" + value = aws_eks_cluster.main.status +} + +output "cluster_iam_role_arn" { + description = "IAM role ARN of the EKS cluster" + value = aws_iam_role.cluster.arn +} + +output "node_group_iam_role_arn" { + description = "IAM role ARN of the EKS node groups" + value = aws_iam_role.node_group.arn +} + +output "node_group_iam_role_name" { + description = "IAM role name of the EKS node groups" + value = aws_iam_role.node_group.name +} + +output "system_node_group_id" { + description = "ID of the system node group. Empty string when use_eks_auto_mode is true." + value = var.use_eks_auto_mode ? "" : aws_eks_node_group.system[0].id +} + +output "system_node_group_status" { + description = "Status of the system node group. Empty string when use_eks_auto_mode is true." + value = var.use_eks_auto_mode ? "" : aws_eks_node_group.system[0].status +} + +output "services_node_group_id" { + description = "ID of the services node group. Empty string when use_eks_auto_mode is true." + value = var.use_eks_auto_mode ? "" : aws_eks_node_group.services[0].id +} + +output "services_node_group_status" { + description = "Status of the services node group. Empty string when use_eks_auto_mode is true." + value = var.use_eks_auto_mode ? "" : aws_eks_node_group.services[0].status +} + +output "brainstore_reader_node_group_id" { + description = "ID of the brainstore reader node group. Empty string when use_eks_auto_mode is true." + value = var.use_eks_auto_mode ? "" : aws_eks_node_group.brainstore_reader[0].id +} + +output "brainstore_reader_node_group_status" { + description = "Status of the brainstore reader node group. Empty string when use_eks_auto_mode is true." + value = var.use_eks_auto_mode ? "" : aws_eks_node_group.brainstore_reader[0].status +} + +output "brainstore_writer_node_group_id" { + description = "ID of the brainstore writer node group. Empty string when use_eks_auto_mode is true." + value = var.use_eks_auto_mode ? "" : aws_eks_node_group.brainstore_writer[0].id +} + +output "brainstore_writer_node_group_status" { + description = "Status of the brainstore writer node group. Empty string when use_eks_auto_mode is true." + value = var.use_eks_auto_mode ? "" : aws_eks_node_group.brainstore_writer[0].status +} + +output "services_spot_node_group_id" { + description = "ID of the services spot node group. Empty string if enable_services_spot_node_group is false." + value = !var.use_eks_auto_mode && var.enable_services_spot_node_group ? aws_eks_node_group.services_spot[0].id : "" +} + +output "services_spot_node_group_status" { + description = "Status of the services spot node group. Empty string if enable_services_spot_node_group is false." + value = !var.use_eks_auto_mode && var.enable_services_spot_node_group ? aws_eks_node_group.services_spot[0].status : "" +} + +output "brainstore_spot_node_group_id" { + description = "ID of the brainstore spot node group. Empty string if disabled or when use_eks_auto_mode is true." + value = !var.use_eks_auto_mode && var.enable_brainstore_spot_node_group ? aws_eks_node_group.brainstore_spot[0].id : "" +} + +output "brainstore_spot_node_group_status" { + description = "Status of the brainstore spot node group. Empty string if disabled or when use_eks_auto_mode is true." + value = !var.use_eks_auto_mode && var.enable_brainstore_spot_node_group ? aws_eks_node_group.brainstore_spot[0].status : "" +} + +output "lb_controller_role_arn" { + description = "IAM role ARN used by the AWS Load Balancer Controller Pod Identity association." + value = aws_iam_role.lb_controller.arn +} + +output "braintrust_api_pod_identity_association_id" { + description = "ID of the Pod Identity association for the Braintrust API service account." + value = aws_eks_pod_identity_association.braintrust_api.association_id +} + +output "brainstore_pod_identity_association_id" { + description = "ID of the Pod Identity association for the Brainstore service account." + value = aws_eks_pod_identity_association.brainstore.association_id +} + +output "nlb_arn" { + description = "ARN of the pre-created internal NLB used by CloudFront VPC Origin." + value = var.enable_cloudfront_nlb_ingress ? aws_lb.api[0].arn : null +} + +output "nlb_name" { + description = "Name of the pre-created internal NLB. Pass to the aws-load-balancer-name service annotation so the controller adopts it." + value = var.enable_cloudfront_nlb_ingress ? aws_lb.api[0].name : null +} + +output "nlb_dns_name" { + description = "DNS name of the pre-created internal NLB." + value = var.enable_cloudfront_nlb_ingress ? aws_lb.api[0].dns_name : null +} + +output "nlb_security_group_id" { + description = "Security group ID attached to the internal NLB." + value = var.enable_cloudfront_nlb_ingress ? aws_security_group.nlb_cloudfront[0].id : null +} + +output "cloudfront_distribution_domain_name" { + description = "CloudFront distribution domain name." + value = var.enable_cloudfront_nlb_ingress ? aws_cloudfront_distribution.dataplane[0].domain_name : null +} + +output "cloudfront_distribution_arn" { + description = "CloudFront distribution ARN." + value = var.enable_cloudfront_nlb_ingress ? aws_cloudfront_distribution.dataplane[0].arn : null +} + +output "cloudfront_distribution_hosted_zone_id" { + description = "CloudFront distribution hosted zone ID." + value = var.enable_cloudfront_nlb_ingress ? aws_cloudfront_distribution.dataplane[0].hosted_zone_id : null +} + +output "api_url" { + description = "Primary Braintrust API URL for this EKS deployment." + value = var.enable_cloudfront_nlb_ingress ? "https://${var.custom_domain != null ? var.custom_domain : aws_cloudfront_distribution.dataplane[0].domain_name}" : null +} diff --git a/modules/eks-cluster/variables.tf b/modules/eks-cluster/variables.tf new file mode 100644 index 00000000..64954f20 --- /dev/null +++ b/modules/eks-cluster/variables.tf @@ -0,0 +1,449 @@ +#---------------------------------------------------------------------------------------------- +# Common +#---------------------------------------------------------------------------------------------- +variable "deployment_name" { + description = "Name of the deployment. Used to prefix resource names." + type = string +} + +variable "custom_tags" { + type = map(string) + description = "Additional tags to apply to all resources" + default = {} +} + +variable "permissions_boundary_arn" { + type = string + description = "IAM permissions boundary ARN applied to IAM roles created by this module." + default = null +} + +#---------------------------------------------------------------------------------------------- +# EKS Cluster +#---------------------------------------------------------------------------------------------- +variable "kubernetes_version" { + type = string + description = "Kubernetes version for the EKS cluster" + default = "1.31" +} + +variable "subnet_ids" { + type = list(string) + description = "List of subnet IDs for the EKS cluster and node groups. Must be in at least two different availability zones." +} + +variable "vpc_id" { + type = string + description = "VPC ID where the EKS cluster, internal NLB, and security groups are created." +} + +variable "aws_load_balancer_controller_service_account" { + type = string + description = "Kubernetes service account name used by the AWS Load Balancer Controller." + default = "aws-load-balancer-controller" +} + +variable "braintrust_namespace" { + type = string + description = "Kubernetes namespace used by Braintrust workloads." +} + +variable "braintrust_api_service_account" { + type = string + description = "Kubernetes service account name for the Braintrust API pods." +} + +variable "brainstore_service_account" { + type = string + description = "Kubernetes service account name for Brainstore pods." +} + +variable "braintrust_api_role_arn" { + type = string + description = "IAM role ARN for the Braintrust API Pod Identity association." +} + +variable "brainstore_role_arn" { + type = string + description = "IAM role ARN for the Brainstore Pod Identity association." +} + +variable "enable_private_access" { + type = bool + description = "Whether the Amazon EKS private API server endpoint is enabled" + default = true +} + +variable "enable_public_access" { + type = bool + description = "Whether the Amazon EKS public API server endpoint is enabled" + default = true +} + +variable "public_access_cidrs" { + type = list(string) + description = "List of CIDR blocks that can access the Amazon EKS public API server endpoint" + default = ["0.0.0.0/0"] +} + +variable "eks_access_entries" { + description = "Additional EKS access entries to create for human or CI access. The cluster creator still receives bootstrap admin permissions." + type = map(object({ + principal_arn = string + type = optional(string, "STANDARD") + kubernetes_groups = optional(list(string)) + user_name = optional(string) + policy_associations = optional(map(object({ + policy_arn = string + access_scope = object({ + type = string + namespaces = optional(list(string), []) + }) + })), {}) + })) + default = {} + + validation { + condition = alltrue(flatten([ + for _, entry in var.eks_access_entries : [ + for _, policy_association in entry.policy_associations : ( + contains(["cluster", "namespace"], policy_association.access_scope.type) && + ( + ( + policy_association.access_scope.type == "cluster" && + length(policy_association.access_scope.namespaces) == 0 + ) || + ( + policy_association.access_scope.type == "namespace" && + length(policy_association.access_scope.namespaces) > 0 + ) + ) + ) + ] + ])) + error_message = "Each EKS access policy association must use access_scope.type of cluster with no namespaces, or namespace with at least one namespace." + } +} + +variable "additional_security_group_ids" { + type = list(string) + description = "Additional security group IDs to attach to the EKS cluster" + default = [] +} + +variable "kms_key_arn" { + type = string + description = "ARN of the KMS key to use for encrypting Kubernetes secrets" +} + +variable "cluster_log_types" { + type = list(string) + description = "List of control plane logging types to enable. Valid values: api, audit, authenticator, controllerManager, scheduler" + default = ["api", "audit", "authenticator"] +} + +variable "cluster_log_retention_days" { + type = number + description = "Number of days to retain EKS control plane logs in CloudWatch. Set to 0 for indefinite retention." + default = 90 + validation { + condition = contains([ + 0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, + 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653 + ], var.cluster_log_retention_days) + error_message = "cluster_log_retention_days must be 0 or a valid CloudWatch Logs retention value." + } +} + +#---------------------------------------------------------------------------------------------- +# Node Groups +#---------------------------------------------------------------------------------------------- +variable "enable_node_ssm" { + type = bool + description = "Enable AWS Systems Manager Session Manager for node debugging" + default = false +} + +variable "node_group_ami_type" { + type = string + description = "AMI type for the system and services node groups. Use AL2023_ARM_64_STANDARD if those groups also run Graviton instances." + default = "AL2023_x86_64_STANDARD" + validation { + condition = contains(["AL2023_x86_64_STANDARD", "AL2023_ARM_64_STANDARD", "AL2_x86_64", "AL2_ARM_64"], var.node_group_ami_type) + error_message = "Must be one of: AL2023_x86_64_STANDARD, AL2023_ARM_64_STANDARD, AL2_x86_64, AL2_ARM_64." + } +} + +variable "brainstore_node_group_ami_type" { + type = string + description = "AMI type for the brainstore node groups. Defaults to AL2023_ARM_64_STANDARD because c8gd (Graviton4 NVMe) instances are the recommended default for Brainstore." + default = "AL2023_ARM_64_STANDARD" + validation { + condition = contains(["AL2023_x86_64_STANDARD", "AL2023_ARM_64_STANDARD", "AL2_x86_64", "AL2_ARM_64"], var.brainstore_node_group_ami_type) + error_message = "Must be one of: AL2023_x86_64_STANDARD, AL2023_ARM_64_STANDARD, AL2_x86_64, AL2_ARM_64." + } +} + +# System Node Group +variable "system_node_group_instance_type" { + type = string + description = "Instance type for the system node group" + default = "t3.medium" +} + +variable "system_node_group_desired_size" { + type = number + description = "Desired number of nodes in the system node group" + default = 2 +} + +variable "system_node_group_min_size" { + type = number + description = "Minimum number of nodes in the system node group" + default = 2 +} + +variable "system_node_group_max_size" { + type = number + description = "Maximum number of nodes in the system node group" + default = 4 +} + +variable "system_node_group_disk_size" { + type = number + description = "Root EBS volume size in GB for system node group instances" + default = 50 +} + +# Services Node Group +variable "services_node_group_instance_type" { + type = string + description = "Instance type for the services node group" + default = "r6i.2xlarge" +} + +variable "services_node_group_desired_size" { + type = number + description = "Desired number of nodes in the services node group" + default = 2 +} + +variable "services_node_group_min_size" { + type = number + description = "Minimum number of nodes in the services node group" + default = 2 +} + +variable "services_node_group_max_size" { + type = number + description = "Maximum number of nodes in the services node group" + default = 10 +} + +variable "services_node_group_disk_size" { + type = number + description = "Root EBS volume size in GB for services node group instances" + default = 100 +} + +# Brainstore Reader Node Group (reader + fast-reader pods) +variable "brainstore_reader_node_group_instance_type" { + type = string + description = "Instance type for the brainstore reader node group. Must be an instance with local NVMe storage. c8gd (Graviton4) instances are the recommended default." + default = "c8gd.8xlarge" +} + +variable "brainstore_reader_node_group_desired_size" { + type = number + description = "Desired number of nodes in the brainstore reader node group" + default = 2 +} + +variable "brainstore_reader_node_group_min_size" { + type = number + description = "Minimum number of nodes in the brainstore reader node group" + default = 2 +} + +variable "brainstore_reader_node_group_max_size" { + type = number + description = "Maximum number of nodes in the brainstore reader node group" + default = 10 +} + +# Brainstore Writer Node Group +variable "brainstore_writer_node_group_instance_type" { + type = string + description = "Instance type for the brainstore writer node group. Must be an instance with local NVMe storage. Defaults to c8gd.16xlarge (2× the vCPU/memory of the reader default)." + default = "c8gd.16xlarge" +} + +variable "brainstore_writer_node_group_desired_size" { + type = number + description = "Desired number of nodes in the brainstore writer node group" + default = 1 +} + +variable "brainstore_writer_node_group_min_size" { + type = number + description = "Minimum number of nodes in the brainstore writer node group" + default = 1 +} + +variable "brainstore_writer_node_group_max_size" { + type = number + description = "Maximum number of nodes in the brainstore writer node group" + default = 5 +} + +# Spot Node Groups +# When enabled, a spot node group is created alongside the on-demand baseline. The on-demand +# group provides one stable node at all times; the spot group adds optional extra capacity for +# non-Auto-Mode deployments. +variable "enable_services_spot_node_group" { + type = bool + description = "Add a spot node group for services burst capacity. The on-demand node group remains the stable baseline; spot provides optional extra capacity for non-Auto-Mode deployments." + default = false +} + +variable "services_spot_node_group_instance_types" { + type = list(string) + description = "Instance types for the services spot node group. Providing multiple similar types improves spot availability. All listed types should have the same CPU and memory ratio." + default = ["r6i.2xlarge", "r5.2xlarge", "r6a.2xlarge"] +} + +variable "services_spot_node_group_min_size" { + type = number + description = "Minimum number of nodes in the services spot node group. Defaults to 0 so the group scales to zero when idle." + default = 0 +} + +variable "services_spot_node_group_max_size" { + type = number + description = "Maximum number of nodes in the services spot node group." + default = 9 +} + +variable "enable_brainstore_spot_node_group" { + type = bool + description = "Add a spot node group for brainstore burst capacity. Note: spot interruptions cause cache loss; pods rehydrate from S3 on restart." + default = false +} + +variable "brainstore_spot_node_group_instance_types" { + type = list(string) + description = "Instance types for the brainstore spot node group. Must be NVMe-backed Graviton instances so the local NVMe mount script runs correctly. All listed types must be ARM64-compatible." + default = ["c8gd.8xlarge", "c8gd.12xlarge", "c6gd.8xlarge"] +} + +variable "brainstore_spot_node_group_min_size" { + type = number + description = "Minimum number of nodes in the brainstore spot node group. Defaults to 0 so the group scales to zero when idle." + default = 0 +} + +variable "brainstore_spot_node_group_max_size" { + type = number + description = "Maximum number of nodes in the brainstore spot node group." + default = 9 +} + +variable "enable_brainstore_writer_spot_node_group" { + type = bool + description = "Add a spot node group for brainstore writer burst capacity. Spot interruptions cause cache loss; pods rehydrate from S3 on restart." + default = false +} + +variable "brainstore_writer_spot_node_group_instance_types" { + type = list(string) + description = "Instance types for the brainstore writer spot node group. Must be NVMe-backed Graviton instances with 2× the vCPU/memory of reader instances." + default = ["c8gd.16xlarge", "c7gd.16xlarge", "c6gd.16xlarge"] +} + +variable "brainstore_writer_spot_node_group_min_size" { + type = number + description = "Minimum number of nodes in the brainstore writer spot node group." + default = 0 +} + +variable "brainstore_writer_spot_node_group_max_size" { + type = number + description = "Maximum number of nodes in the brainstore writer spot node group." + default = 9 +} + +#---------------------------------------------------------------------------------------------- +# EKS Addons +#---------------------------------------------------------------------------------------------- +variable "vpc_cni_addon_version" { + type = string + description = "Version of the VPC CNI addon. If not specified, uses the default version for the cluster." + default = null +} + +variable "kube_proxy_addon_version" { + type = string + description = "Version of the kube-proxy addon. If not specified, uses the default version for the cluster." + default = null +} + +variable "coredns_addon_version" { + type = string + description = "Version of the CoreDNS addon. If not specified, uses the default version for the cluster." + default = null +} + + +variable "pod_identity_addon_version" { + type = string + description = "Version of the EKS Pod Identity agent addon. If not specified, uses the default version for the cluster." + default = null +} + +#---------------------------------------------------------------------------------------------- +# Ingress Edge +#---------------------------------------------------------------------------------------------- +variable "enable_cloudfront_nlb_ingress" { + type = bool + description = "When true, creates the module-managed CloudFront + private NLB ingress stack for the EKS deployment. Set false if you want the EKS cluster without the bundled ingress so you can bring your own ingress/controller setup." + default = true +} + +variable "cloudfront_price_class" { + type = string + description = "CloudFront price class for the EKS API edge." + default = "PriceClass_100" +} + +variable "custom_domain" { + type = string + description = "Optional custom domain name for the CloudFront distribution." + default = null +} + +variable "custom_certificate_arn" { + type = string + description = "Optional ACM certificate ARN in us-east-1 for the CloudFront custom domain." + default = null +} + +variable "waf_acl_id" { + type = string + description = "Optional WAF Web ACL ID/ARN to associate with the CloudFront distribution." + default = null +} + +variable "use_global_ai_proxy" { + type = bool + description = "Whether to route proxy, eval, and function edge paths to the global Cloudflare proxy instead of the EKS API origin." + default = false +} + +#---------------------------------------------------------------------------------------------- +# EKS Auto Mode +#---------------------------------------------------------------------------------------------- +variable "use_eks_auto_mode" { + type = bool + description = "Enable EKS Auto Mode. When true, EKS manages compute (system and services nodes), load balancing, and block storage automatically. System and services managed node groups are not created — EKS Auto Mode provisions them via built-in node pools. Brainstore capacity is supplied separately through custom Auto Mode NodeClass and NodePool resources created by the eks-deploy module." + default = true +} diff --git a/modules/eks-cluster/versions.tf b/modules/eks-cluster/versions.tf new file mode 100644 index 00000000..208a27a0 --- /dev/null +++ b/modules/eks-cluster/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.10" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + } +} diff --git a/modules/eks-deploy/charts/brainstore-auto-mode/Chart.yaml b/modules/eks-deploy/charts/brainstore-auto-mode/Chart.yaml new file mode 100644 index 00000000..270d2c7e --- /dev/null +++ b/modules/eks-deploy/charts/brainstore-auto-mode/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: brainstore-auto-mode +description: Brainstore EKS Auto Mode NodeClass and NodePool resources. +type: application +version: 0.1.0 +appVersion: "1.0.0" diff --git a/modules/eks-deploy/charts/brainstore-auto-mode/templates/reader-nodeclass.yaml b/modules/eks-deploy/charts/brainstore-auto-mode/templates/reader-nodeclass.yaml new file mode 100644 index 00000000..1d43d1cd --- /dev/null +++ b/modules/eks-deploy/charts/brainstore-auto-mode/templates/reader-nodeclass.yaml @@ -0,0 +1,14 @@ +apiVersion: eks.amazonaws.com/v1 +kind: NodeClass +metadata: + name: {{ .Values.reader.nodeClassName }} +spec: + role: {{ .Values.nodeRoleName | quote }} + ephemeralStorage: + size: {{ .Values.reader.ephemeralStorageSize | quote }} + subnetSelectorTerms: + - tags: + {{ .Values.subnetTagKey }}: {{ .Values.subnetTagValue | quote }} + securityGroupSelectorTerms: + - tags: + {{ .Values.securityGroupKey }}: {{ .Values.securityGroupValue | quote }} diff --git a/modules/eks-deploy/charts/brainstore-auto-mode/templates/reader-nodepool.yaml b/modules/eks-deploy/charts/brainstore-auto-mode/templates/reader-nodepool.yaml new file mode 100644 index 00000000..2e61324a --- /dev/null +++ b/modules/eks-deploy/charts/brainstore-auto-mode/templates/reader-nodepool.yaml @@ -0,0 +1,35 @@ +apiVersion: karpenter.sh/v1 +kind: NodePool +metadata: + name: {{ .Values.reader.nodePoolName }} +spec: + template: + metadata: + labels: + role: brainstore-reader + spec: + taints: + - key: dedicated + value: brainstore + effect: NoSchedule + requirements: + - key: eks.amazonaws.com/instance-family + operator: In + values: +{{ toYaml .Values.instanceFamilies | indent 12 }} + - key: eks.amazonaws.com/instance-size + operator: In + values: +{{ toYaml .Values.reader.instanceSizes | indent 12 }} + - key: karpenter.sh/capacity-type + operator: In + values: + - on-demand + - spot + nodeClassRef: + group: eks.amazonaws.com + kind: NodeClass + name: {{ .Values.reader.nodeClassName }} + limits: + cpu: {{ .Values.reader.cpuLimit | quote }} + memory: {{ .Values.reader.memoryLimit | quote }} diff --git a/modules/eks-deploy/charts/brainstore-auto-mode/templates/writer-nodeclass.yaml b/modules/eks-deploy/charts/brainstore-auto-mode/templates/writer-nodeclass.yaml new file mode 100644 index 00000000..2b12b96d --- /dev/null +++ b/modules/eks-deploy/charts/brainstore-auto-mode/templates/writer-nodeclass.yaml @@ -0,0 +1,14 @@ +apiVersion: eks.amazonaws.com/v1 +kind: NodeClass +metadata: + name: {{ .Values.writer.nodeClassName }} +spec: + role: {{ .Values.nodeRoleName | quote }} + ephemeralStorage: + size: {{ .Values.writer.ephemeralStorageSize | quote }} + subnetSelectorTerms: + - tags: + {{ .Values.subnetTagKey }}: {{ .Values.subnetTagValue | quote }} + securityGroupSelectorTerms: + - tags: + {{ .Values.securityGroupKey }}: {{ .Values.securityGroupValue | quote }} diff --git a/modules/eks-deploy/charts/brainstore-auto-mode/templates/writer-nodepool.yaml b/modules/eks-deploy/charts/brainstore-auto-mode/templates/writer-nodepool.yaml new file mode 100644 index 00000000..ab93e2a0 --- /dev/null +++ b/modules/eks-deploy/charts/brainstore-auto-mode/templates/writer-nodepool.yaml @@ -0,0 +1,35 @@ +apiVersion: karpenter.sh/v1 +kind: NodePool +metadata: + name: {{ .Values.writer.nodePoolName }} +spec: + template: + metadata: + labels: + role: brainstore-writer + spec: + taints: + - key: dedicated + value: brainstore + effect: NoSchedule + requirements: + - key: eks.amazonaws.com/instance-family + operator: In + values: +{{ toYaml .Values.instanceFamilies | indent 12 }} + - key: eks.amazonaws.com/instance-size + operator: In + values: +{{ toYaml .Values.writer.instanceSizes | indent 12 }} + - key: karpenter.sh/capacity-type + operator: In + values: + - on-demand + - spot + nodeClassRef: + group: eks.amazonaws.com + kind: NodeClass + name: {{ .Values.writer.nodeClassName }} + limits: + cpu: {{ .Values.writer.cpuLimit | quote }} + memory: {{ .Values.writer.memoryLimit | quote }} diff --git a/modules/eks-deploy/main.tf b/modules/eks-deploy/main.tf new file mode 100644 index 00000000..fda55e91 --- /dev/null +++ b/modules/eks-deploy/main.tf @@ -0,0 +1,107 @@ +data "aws_region" "current" {} + +resource "kubernetes_namespace" "braintrust" { + metadata { + name = var.namespace + } +} + +resource "kubernetes_secret" "braintrust" { + metadata { + name = "braintrust-secrets" + namespace = var.namespace + } + + data = { + PG_URL = "postgresql://${var.postgres_username}:${var.postgres_password}@${var.postgres_host}:${var.postgres_port}/postgres?sslmode=require" + REDIS_URL = "redis://${var.redis_host}:${var.redis_port}" + BRAINSTORE_LICENSE_KEY = var.brainstore_license_key + FUNCTION_SECRET_KEY = var.function_secret_key + } + + type = "Opaque" + + depends_on = [kubernetes_namespace.braintrust] +} + +resource "helm_release" "aws_load_balancer_controller" { + name = "aws-load-balancer-controller" + repository = "https://aws.github.io/eks-charts" + chart = "aws-load-balancer-controller" + version = var.aws_load_balancer_controller_chart_version + namespace = "kube-system" + + values = [ + yamlencode({ + clusterName = var.cluster_name + region = data.aws_region.current.region + vpcId = var.vpc_id + serviceAccount = { + create = true + name = var.aws_load_balancer_controller_service_account + } + }) + ] +} + +resource "helm_release" "brainstore_auto_mode" { + count = var.use_auto_mode ? 1 : 0 + + name = "brainstore-auto-mode" + chart = "${path.module}/charts/brainstore-auto-mode" + namespace = var.namespace + create_namespace = false + wait = false + + values = [ + yamlencode({ + nodeRoleName = var.node_role_name + clusterName = var.cluster_name + instanceFamilies = var.brainstore_instance_families + subnetTagKey = "kubernetes.io/role/internal-elb" + subnetTagValue = "1" + securityGroupKey = "kubernetes.io/cluster/${var.cluster_name}" + securityGroupValue = "owned" + reader = { + nodeClassName = "brainstore-reader" + nodePoolName = "brainstore-reader" + ephemeralStorageSize = "1000Gi" + instanceSizes = var.brainstore_reader_instance_sizes + cpuLimit = "80" + memoryLimit = "160Gi" + } + writer = { + nodeClassName = "brainstore-writer" + nodePoolName = "brainstore-writer" + ephemeralStorageSize = "2000Gi" + instanceSizes = var.brainstore_writer_instance_sizes + cpuLimit = "128" + memoryLimit = "256Gi" + } + }) + ] + + depends_on = [kubernetes_namespace.braintrust] +} + +resource "helm_release" "braintrust" { + count = var.manage_braintrust_helm_release ? 1 : 0 + + name = "braintrust" + repository = "oci://public.ecr.aws/braintrust/helm" + chart = "braintrust" + version = var.helm_chart_version + namespace = var.namespace + create_namespace = false + timeout = 1200 + cleanup_on_fail = true + wait = false + + values = local.helm_values_documents + + depends_on = [ + helm_release.aws_load_balancer_controller, + helm_release.brainstore_auto_mode, + kubernetes_secret.braintrust, + ] +} diff --git a/modules/eks-deploy/outputs.tf b/modules/eks-deploy/outputs.tf new file mode 100644 index 00000000..d1d5c4d3 --- /dev/null +++ b/modules/eks-deploy/outputs.tf @@ -0,0 +1,24 @@ +output "namespace" { + value = kubernetes_namespace.braintrust.metadata[0].name + description = "Namespace the Braintrust workloads were deployed into." +} + +output "braintrust_release_name" { + value = "braintrust" + description = "Name to use for the Braintrust Helm release." +} + +output "braintrust_helm_release_managed" { + value = var.manage_braintrust_helm_release + description = "True when Terraform manages the Braintrust Helm release." +} + +output "braintrust_helm_values_yaml" { + value = local.helm_values_yaml + description = "Generated multi-document Helm values for the Braintrust chart. Write to a file and pass with --values when doing a manual helm install/upgrade." +} + +output "aws_load_balancer_controller_release_name" { + value = helm_release.aws_load_balancer_controller.name + description = "Name of the AWS Load Balancer Controller Helm release." +} diff --git a/modules/eks-deploy/overrides.tf b/modules/eks-deploy/overrides.tf new file mode 100644 index 00000000..c48b50dd --- /dev/null +++ b/modules/eks-deploy/overrides.tf @@ -0,0 +1,148 @@ +locals { + api_toleration = { + key = "dedicated" + operator = "Equal" + value = "services" + effect = "NoSchedule" + } + + brainstore_toleration = { + key = "dedicated" + operator = "Equal" + value = "brainstore" + effect = "NoSchedule" + } + + brainstore_reader_node_selector = var.use_auto_mode ? tomap({ + "karpenter.sh/nodepool" = "brainstore-reader" + }) : tomap({ + role = "brainstore-reader" + }) + + brainstore_writer_node_selector = var.use_auto_mode ? tomap({ + "karpenter.sh/nodepool" = "brainstore-writer" + }) : tomap({ + role = "brainstore-writer" + }) + + api_base = { + service = { + type = "LoadBalancer" + } + annotations = { + service = merge( + { + "service.beta.kubernetes.io/aws-load-balancer-scheme" = "internal" + "service.beta.kubernetes.io/aws-load-balancer-type" = "external" + "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type" = "instance" + "service.beta.kubernetes.io/aws-load-balancer-security-groups" = var.nlb_security_group_id + "service.beta.kubernetes.io/aws-load-balancer-name" = var.nlb_name + }, + var.prepare_for_destroy ? { + # Shorten target deregistration so destroy is less likely to stall on + # AWS Load Balancer Controller finalizers. + "service.beta.kubernetes.io/aws-load-balancer-target-group-attributes" = "deregistration_delay.timeout_seconds=0" + } : {} + ) + } + serviceAccount = { + name = var.api_service_account_name + awsRoleArn = var.api_handler_role_arn + } + } + + # Emitted as a separate Helm values layer so compact() drops it cleanly in Auto Mode, + # avoiding Terraform object type-consistency errors from conditional merges. + api_scheduling_yaml = var.use_auto_mode ? "" : yamlencode({ + api = { + nodeSelector = { role = "services" } + tolerations = [local.api_toleration] + } + }) + + brainstore_base = { + serviceAccount = { + name = var.brainstore_service_account_name + awsRoleArn = var.brainstore_iam_role_arn + } + reader = { + nodeSelector = local.brainstore_reader_node_selector + tolerations = [local.brainstore_toleration] + } + fastreader = { + nodeSelector = local.brainstore_reader_node_selector + tolerations = [local.brainstore_toleration] + } + writer = { + nodeSelector = local.brainstore_writer_node_selector + tolerations = [local.brainstore_toleration] + } + } + + api_structured_override = merge( + var.api_helm.replicas != null ? { replicas = var.api_helm.replicas } : {}, + var.api_helm.resources != null ? { resources = var.api_helm.resources } : {}, + ) + + reader_structured_override = merge( + var.brainstore_reader_helm.replicas != null ? { replicas = var.brainstore_reader_helm.replicas } : {}, + var.brainstore_reader_helm.resources != null ? { resources = var.brainstore_reader_helm.resources } : {}, + ) + + fastreader_structured_override = merge( + var.brainstore_fastreader_helm.replicas != null ? { replicas = var.brainstore_fastreader_helm.replicas } : {}, + var.brainstore_fastreader_helm.resources != null ? { resources = var.brainstore_fastreader_helm.resources } : {}, + ) + + writer_structured_override = merge( + var.brainstore_writer_helm.replicas != null ? { replicas = var.brainstore_writer_helm.replicas } : {}, + var.brainstore_writer_helm.resources != null ? { resources = var.brainstore_writer_helm.resources } : {}, + ) + + brainstore_structured_override = merge( + length(local.reader_structured_override) > 0 ? { reader = local.reader_structured_override } : {}, + length(local.fastreader_structured_override) > 0 ? { fastreader = local.fastreader_structured_override } : {}, + length(local.writer_structured_override) > 0 ? { writer = local.writer_structured_override } : {}, + ) + + helm_structured_overrides = merge( + length(local.api_structured_override) > 0 ? { api = local.api_structured_override } : {}, + length(local.brainstore_structured_override) > 0 ? { brainstore = local.brainstore_structured_override } : {}, + ) + + helm_base_values = { + global = { + orgName = var.braintrust_org_name + createNamespace = false + namespace = var.namespace + } + + cloud = "aws" + + skipPgForBrainstoreObjects = var.skip_pg_for_brainstore_objects + brainstoreWalFooterVersion = var.brainstore_wal_footer_version + + objectStorage = { + aws = { + brainstoreBucket = var.brainstore_bucket_name + responseBucket = var.response_bucket_name + codeBundleBucket = var.code_bundle_bucket_name + } + } + + api = local.api_base + brainstore = local.brainstore_base + } + + helm_values_documents = compact([ + local.helm_base_values_yaml, + local.api_scheduling_yaml, + local.helm_structured_overrides_yaml, + var.helm_chart_extra_values, + ]) + + helm_values_yaml = join("\n---\n", local.helm_values_documents) + + helm_base_values_yaml = yamlencode(local.helm_base_values) + helm_structured_overrides_yaml = length(local.helm_structured_overrides) > 0 ? yamlencode(local.helm_structured_overrides) : "" +} diff --git a/modules/eks-deploy/variables.tf b/modules/eks-deploy/variables.tf new file mode 100644 index 00000000..c37744a8 --- /dev/null +++ b/modules/eks-deploy/variables.tf @@ -0,0 +1,253 @@ +variable "deployment_name" { + type = string + description = "Name of this deployment. Used in resource naming and tags." +} + +variable "custom_tags" { + type = map(string) + description = "Custom tags applied to created resources." + default = {} +} + +variable "braintrust_org_name" { + type = string + description = "Braintrust org name. This becomes global.orgName in the Helm chart." +} + +variable "namespace" { + type = string + description = "Kubernetes namespace for Braintrust workloads. Created by this module." +} + +variable "cluster_name" { + type = string + description = "EKS cluster name." +} + +variable "vpc_id" { + type = string + description = "VPC ID used by the AWS Load Balancer Controller Helm release." +} + +variable "nlb_security_group_id" { + type = string + description = "Security group ID attached to the pre-created internal NLB." +} + +variable "nlb_name" { + type = string + description = "Name of the pre-created internal NLB for the Braintrust API service to adopt." +} + +variable "api_handler_role_arn" { + type = string + description = "IAM role ARN consumed by the Braintrust API service account." +} + +variable "api_service_account_name" { + type = string + description = "Kubernetes service account name used by Braintrust API pods." + default = "braintrust-api" +} + +variable "brainstore_iam_role_arn" { + type = string + description = "IAM role ARN consumed by the Brainstore service account." +} + +variable "brainstore_service_account_name" { + type = string + description = "Kubernetes service account name used by Brainstore pods." + default = "brainstore" +} + +variable "brainstore_bucket_name" { + type = string + description = "S3 bucket name for Brainstore data." +} + +variable "response_bucket_name" { + type = string + description = "S3 bucket name for API responses." +} + +variable "code_bundle_bucket_name" { + type = string + description = "S3 bucket name for code bundles." +} + +variable "postgres_host" { + type = string + description = "Postgres host." +} + +variable "postgres_port" { + type = number + description = "Postgres port." +} + +variable "postgres_username" { + type = string + description = "Postgres username." +} + +variable "postgres_password" { + type = string + description = "Postgres password." + sensitive = true +} + +variable "redis_host" { + type = string + description = "Redis host." +} + +variable "redis_port" { + type = number + description = "Redis port." +} + +variable "brainstore_license_key" { + type = string + description = "Brainstore license key." + sensitive = true +} + +variable "function_secret_key" { + type = string + description = "Function secret key stored in the braintrust-secrets Kubernetes secret." + sensitive = true +} + +variable "brainstore_wal_footer_version" { + type = string + description = "WAL footer version passed to the chart." +} + +variable "skip_pg_for_brainstore_objects" { + type = string + description = "Controls which object types bypass PostgreSQL." +} + +variable "helm_chart_version" { + type = string + description = "Version of the Braintrust Helm chart to deploy." +} + +variable "manage_braintrust_helm_release" { + type = bool + description = "When true, Terraform manages the Braintrust Helm release. When false, Terraform still prepares the namespace, secret, controller, and optional Auto Mode resources, but expects a manual helm install/upgrade for the Braintrust chart." + default = true +} + +variable "aws_load_balancer_controller_chart_version" { + type = string + description = "Optional AWS Load Balancer Controller Helm chart version." + default = null +} + +variable "aws_load_balancer_controller_service_account" { + type = string + description = "Service account name used by the AWS Load Balancer Controller." + default = "aws-load-balancer-controller" +} + +variable "api_helm" { + type = object({ + replicas = optional(number) + resources = optional(object({ + requests = object({ cpu = string, memory = string }) + limits = object({ cpu = string, memory = string }) + })) + }) + default = {} + description = "Override replicas and/or resources for the API component." +} + +variable "brainstore_reader_helm" { + type = object({ + replicas = optional(number) + resources = optional(object({ + requests = object({ cpu = string, memory = string }) + limits = object({ cpu = string, memory = string }) + })) + }) + default = {} + description = "Override replicas and/or resources for the Brainstore reader." +} + +variable "brainstore_fastreader_helm" { + type = object({ + replicas = optional(number) + resources = optional(object({ + requests = object({ cpu = string, memory = string }) + limits = object({ cpu = string, memory = string }) + })) + }) + default = {} + description = "Override replicas and/or resources for the Brainstore fast reader." +} + +variable "brainstore_writer_helm" { + type = object({ + replicas = optional(number) + resources = optional(object({ + requests = object({ cpu = string, memory = string }) + limits = object({ cpu = string, memory = string }) + })) + }) + default = {} + description = "Override replicas and/or resources for the Brainstore writer." +} + +variable "helm_chart_extra_values" { + type = string + description = "Escape-hatch YAML appended to the chart values. Later values win." + default = "" +} + +variable "prepare_for_destroy" { + type = bool + description = "When true, reduces AWS load balancer target deregistration delay for the Braintrust API Service before destroy." + default = false +} + +variable "use_auto_mode" { + type = bool + description = "When true, omits node selectors and tolerations for managed node groups that do not exist in Auto Mode, and deploys Brainstore NodeClass/NodePool resources instead." + default = false +} + +variable "node_role_name" { + type = string + description = "IAM role name for Auto Mode nodes. Required when use_auto_mode is true — used as the NodeClass role reference." + default = "" +} + +variable "brainstore_instance_families" { + type = list(string) + description = "EC2 instance families for Brainstore NodePools. Must be NVMe-backed Graviton families (c7gd, c8gd)." + default = ["c7gd", "c8gd"] +} + +variable "brainstore_reader_instance_sizes" { + type = list(string) + description = "EC2 instance sizes allowed for the Brainstore reader and fast-reader Auto Mode NodePool." + default = ["4xlarge"] + + validation { + condition = length(var.brainstore_reader_instance_sizes) > 0 + error_message = "brainstore_reader_instance_sizes must contain at least one size." + } +} + +variable "brainstore_writer_instance_sizes" { + type = list(string) + description = "EC2 instance sizes allowed for the Brainstore writer Auto Mode NodePool." + default = ["8xlarge"] + + validation { + condition = length(var.brainstore_writer_instance_sizes) > 0 + error_message = "brainstore_writer_instance_sizes must contain at least one size." + } +} diff --git a/modules/eks-deploy/versions.tf b/modules/eks-deploy/versions.tf new file mode 100644 index 00000000..88d6eb0f --- /dev/null +++ b/modules/eks-deploy/versions.tf @@ -0,0 +1,18 @@ +terraform { + required_version = ">= 1.10.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = "~> 2.0" + } + helm = { + source = "hashicorp/helm" + version = "~> 2.0" + } + } +} diff --git a/modules/services-common/iam-quarantine.tf b/modules/services-common/iam-quarantine.tf index a1c00ff3..9d21d876 100644 --- a/modules/services-common/iam-quarantine.tf +++ b/modules/services-common/iam-quarantine.tf @@ -1,6 +1,6 @@ # Quarantine VPC IAM resources # These resources are needed for the Quarantine VPC functionality and are deployed -# even when use_deployment_mode_external_eks = true, as they are required for +# even when EKS deployment mode is enabled, as they are required for # running user-defined functions in an isolated environment. # The role used by the API handler to invoke the user-defined quarantined functions diff --git a/modules/services-common/main.tf b/modules/services-common/main.tf index ae966871..eeee7fdf 100644 --- a/modules/services-common/main.tf +++ b/modules/services-common/main.tf @@ -3,8 +3,13 @@ locals { BraintrustDeploymentName = var.deployment_name }, var.custom_tags) - # OIDC issuer URL for EKS cluster. This is used when optionally enabling IRSA (Identity Role for Service Accounts) - eks_oidc_issuer_url = var.eks_cluster_arn != null && var.enable_eks_irsa ? data.aws_eks_cluster.cluster[0].identity[0].oidc[0].issuer : null + # Prefer an explicitly provided OIDC issuer URL. For externally managed EKS + # clusters, fall back to looking up the cluster only when IRSA is enabled and + # the caller did not pass the issuer URL directly. + eks_oidc_issuer_url = var.enable_eks_irsa ? coalesce( + var.eks_cluster_oidc_issuer_url, + try(data.aws_eks_cluster.cluster[0].identity[0].oidc[0].issuer, null) + ) : null } data "aws_region" "current" {} @@ -12,6 +17,6 @@ data "aws_region" "current" {} data "aws_caller_identity" "current" {} data "aws_eks_cluster" "cluster" { - count = var.eks_cluster_arn != null && var.enable_eks_irsa ? 1 : 0 + count = var.enable_eks_irsa && var.lookup_eks_cluster_oidc_issuer_url ? 1 : 0 name = split("/", var.eks_cluster_arn)[1] } diff --git a/modules/services-common/variables.tf b/modules/services-common/variables.tf index 8c434770..67bb02da 100644 --- a/modules/services-common/variables.tf +++ b/modules/services-common/variables.tf @@ -35,6 +35,23 @@ variable "eks_cluster_arn" { default = null } +variable "eks_cluster_oidc_issuer_url" { + type = string + description = "Optional. OIDC issuer URL for the EKS cluster. Used when enable_eks_irsa is true." + default = null +} + +variable "lookup_eks_cluster_oidc_issuer_url" { + type = bool + description = "When true, resolve the EKS cluster OIDC issuer URL from aws_eks_cluster using eks_cluster_arn. Leave false when the caller already passes eks_cluster_oidc_issuer_url, especially if the cluster is being created in the same apply." + default = false + + validation { + condition = !var.lookup_eks_cluster_oidc_issuer_url || var.eks_cluster_arn != null + error_message = "eks_cluster_arn is required when lookup_eks_cluster_oidc_issuer_url is true." + } +} + variable "eks_namespace" { type = string description = "Optional. If you're using EKS with IRSA or EKS Pod Identity, this restricts the IAM roles to this namespace." diff --git a/modules/services/outputs.tf b/modules/services/outputs.tf index 9a86fd54..9fb66444 100644 --- a/modules/services/outputs.tf +++ b/modules/services/outputs.tf @@ -24,7 +24,7 @@ output "catchup_etl_arn" { } output "quarantine_warmup_arn" { - description = "The ARN of the quarantine warmup lambda function. Only created when use_deployment_mode_external_eks is false." + description = "The ARN of the quarantine warmup lambda function. Only created when EKS deployment mode is disabled." value = var.use_quarantine_vpc ? aws_lambda_function.quarantine_warmup[0].arn : null } @@ -42,4 +42,3 @@ output "ai_proxy_url_ssm_arn" { description = "The ARN of the SSM parameter containing the AI proxy URL" value = aws_ssm_parameter.ai_proxy_url.arn } - diff --git a/modules/storage/s3-brainstore.tf b/modules/storage/s3-brainstore.tf index aa2021d7..b0e6e6d7 100644 --- a/modules/storage/s3-brainstore.tf +++ b/modules/storage/s3-brainstore.tf @@ -69,7 +69,7 @@ resource "aws_s3_bucket_lifecycle_configuration" "brainstore" { expiration { # We use the same var for the expiration interval and the delete interval - # in cleanup-old-versions so the total time to deletion is 2 * var.s3_bucket_retention_days + # in cleanup-old-versions so the total time to deletion is 2 * var.brainstore_s3_bucket_retention_days days = var.brainstore_s3_bucket_retention_days } } @@ -85,7 +85,7 @@ resource "aws_s3_bucket_lifecycle_configuration" "brainstore" { expiration { # We use the same var for the expiration interval and the delete interval - # in cleanup-old-versions so the total time to deletion is 2 * var.s3_bucket_retention_days + # in cleanup-old-versions so the total time to deletion is 2 * var.brainstore_s3_bucket_retention_days days = var.brainstore_s3_bucket_retention_days } } diff --git a/modules/vpc/main.tf b/modules/vpc/main.tf index d706fcce..dc9fe244 100644 --- a/modules/vpc/main.tf +++ b/modules/vpc/main.tf @@ -84,7 +84,8 @@ resource "aws_subnet" "public_subnet_1" { map_public_ip_on_launch = false tags = merge({ - Name = "${var.deployment_name}-${var.vpc_name}-public-subnet-1" + Name = "${var.deployment_name}-${var.vpc_name}-public-subnet-1" + "kubernetes.io/role/elb" = "1" }, local.common_tags) lifecycle { @@ -92,13 +93,40 @@ resource "aws_subnet" "public_subnet_1" { } } +resource "aws_subnet" "public_subnet_2" { + count = var.public_subnet_2_cidr != null ? 1 : 0 + vpc_id = aws_vpc.vpc.id + cidr_block = var.public_subnet_2_cidr + availability_zone = var.public_subnet_2_az + map_public_ip_on_launch = false + + tags = merge({ + Name = "${var.deployment_name}-${var.vpc_name}-public-subnet-2" + "kubernetes.io/role/elb" = "1" + }, local.common_tags) +} + +resource "aws_subnet" "public_subnet_3" { + count = var.public_subnet_3_cidr != null ? 1 : 0 + vpc_id = aws_vpc.vpc.id + cidr_block = var.public_subnet_3_cidr + availability_zone = var.public_subnet_3_az + map_public_ip_on_launch = false + + tags = merge({ + Name = "${var.deployment_name}-${var.vpc_name}-public-subnet-3" + "kubernetes.io/role/elb" = "1" + }, local.common_tags) +} + resource "aws_subnet" "private_subnet_1" { vpc_id = aws_vpc.vpc.id cidr_block = var.private_subnet_1_cidr availability_zone = var.private_subnet_1_az tags = merge({ - Name = "${var.deployment_name}-${var.vpc_name}-private-subnet-1" + Name = "${var.deployment_name}-${var.vpc_name}-private-subnet-1" + "kubernetes.io/role/internal-elb" = "1" }, local.common_tags) lifecycle { @@ -112,7 +140,8 @@ resource "aws_subnet" "private_subnet_2" { availability_zone = var.private_subnet_2_az tags = merge({ - Name = "${var.deployment_name}-${var.vpc_name}-private-subnet-2" + Name = "${var.deployment_name}-${var.vpc_name}-private-subnet-2" + "kubernetes.io/role/internal-elb" = "1" }, local.common_tags) lifecycle { @@ -126,7 +155,8 @@ resource "aws_subnet" "private_subnet_3" { availability_zone = var.private_subnet_3_az tags = merge({ - Name = "${var.deployment_name}-${var.vpc_name}-private-subnet-3" + Name = "${var.deployment_name}-${var.vpc_name}-private-subnet-3" + "kubernetes.io/role/internal-elb" = "1" }, local.common_tags) lifecycle { @@ -154,6 +184,18 @@ resource "aws_route_table_association" "public_subnet_1_association" { subnet_id = aws_subnet.public_subnet_1.id } +resource "aws_route_table_association" "public_subnet_2_association" { + count = var.public_subnet_2_cidr != null ? 1 : 0 + route_table_id = aws_route_table.public_route_table.id + subnet_id = aws_subnet.public_subnet_2[0].id +} + +resource "aws_route_table_association" "public_subnet_3_association" { + count = var.public_subnet_3_cidr != null ? 1 : 0 + route_table_id = aws_route_table.public_route_table.id + subnet_id = aws_subnet.public_subnet_3[0].id +} + resource "aws_vpc_endpoint" "s3" { vpc_id = aws_vpc.vpc.id service_name = "com.amazonaws.${data.aws_region.current.region}.s3" diff --git a/modules/vpc/outputs.tf b/modules/vpc/outputs.tf index eaf7b314..5dcf6b83 100644 --- a/modules/vpc/outputs.tf +++ b/modules/vpc/outputs.tf @@ -9,10 +9,20 @@ output "vpc_cidr" { } output "public_subnet_1_id" { - description = "ID of the public subnet" + description = "ID of the first public subnet" value = aws_subnet.public_subnet_1.id } +output "public_subnet_2_id" { + description = "ID of the second public subnet. Null when public_subnet_2_cidr is not set." + value = var.public_subnet_2_cidr != null ? aws_subnet.public_subnet_2[0].id : null +} + +output "public_subnet_3_id" { + description = "ID of the third public subnet. Null when public_subnet_3_cidr is not set." + value = var.public_subnet_3_cidr != null ? aws_subnet.public_subnet_3[0].id : null +} + output "private_subnet_1_id" { description = "ID of private subnet 1" value = aws_subnet.private_subnet_1.id diff --git a/modules/vpc/variables.tf b/modules/vpc/variables.tf index a4a96b12..7c81ef86 100644 --- a/modules/vpc/variables.tf +++ b/modules/vpc/variables.tf @@ -53,6 +53,30 @@ variable "private_subnet_3_az" { type = string } +variable "public_subnet_2_cidr" { + description = "CIDR block for the second public subnet. If null, the subnet is not created." + type = string + default = null +} + +variable "public_subnet_2_az" { + description = "Availability zone for the second public subnet." + type = string + default = null +} + +variable "public_subnet_3_cidr" { + description = "CIDR block for the third public subnet. If null, the subnet is not created." + type = string + default = null +} + +variable "public_subnet_3_az" { + description = "Availability zone for the third public subnet." + type = string + default = null +} + variable "custom_tags" { description = "Custom tags to apply to all created resources" type = map(string) diff --git a/outputs.tf b/outputs.tf index 0fc056e3..42bfe32f 100644 --- a/outputs.tf +++ b/outputs.tf @@ -15,7 +15,17 @@ output "main_vpc_cidr" { output "main_vpc_public_subnet_1_id" { value = local.main_vpc_public_subnet_1_id - description = "ID of the public subnet in the main VPC" + description = "ID of the first public subnet in the main VPC" +} + +output "main_vpc_public_subnet_2_id" { + value = local.main_vpc_public_subnet_2_id + description = "ID of the second public subnet in the main VPC (null when create_vpc is false)" +} + +output "main_vpc_public_subnet_3_id" { + value = local.main_vpc_public_subnet_3_id + description = "ID of the third public subnet in the main VPC (null when create_vpc is false)" } output "main_vpc_private_subnet_1_id" { @@ -53,6 +63,16 @@ output "brainstore_s3_bucket_name" { description = "Name of the Brainstore S3 bucket" } +output "code_bundle_s3_bucket_name" { + value = module.storage.code_bundle_bucket_id + description = "Name of the S3 bucket used for uploaded code bundles" +} + +output "lambda_responses_s3_bucket_name" { + value = module.storage.lambda_responses_bucket_id + description = "Name of the S3 bucket used for lambda response payloads" +} + output "rds_security_group_id" { value = module.database.rds_security_group_id description = "ID of the security group for the RDS instance" @@ -64,7 +84,7 @@ output "redis_security_group_id" { } output "lambda_security_group_id" { - value = !var.use_deployment_mode_external_eks ? module.services[0].lambda_security_group_id : null + value = !local.use_eks ? module.services[0].lambda_security_group_id : null description = "ID of the security group for the Lambda functions" } @@ -123,24 +143,55 @@ output "redis_arn" { description = "ARN of the Redis instance" } +output "redis_endpoint" { + value = module.redis.redis_endpoint + description = "Hostname of the ElastiCache Redis instance" +} + +output "redis_port" { + value = module.redis.redis_port + description = "Port of the ElastiCache Redis instance" +} + +output "postgres_database_address" { + value = module.database.postgres_database_address + description = "Hostname of the RDS PostgreSQL instance" +} + +output "postgres_database_port" { + value = module.database.postgres_database_port + description = "Port of the RDS PostgreSQL instance" +} + +output "postgres_database_username" { + value = module.database.postgres_database_username + description = "Username of the RDS PostgreSQL instance" +} + +output "postgres_database_password" { + value = module.database.postgres_database_password + description = "Password of the RDS PostgreSQL instance" + sensitive = true +} + output "api_url" { - value = !var.use_deployment_mode_external_eks ? module.ingress[0].api_url : null - description = "The primary endpoint for the dataplane API. This is the value that should be entered into the braintrust dashboard under API URL." + value = !local.use_eks ? module.ingress[0].api_url : (var.create_eks_cluster ? module.eks_cluster[0].api_url : null) + description = "The primary endpoint for the dataplane API. This is the value that should be entered into the Braintrust dashboard under API URL. Null when create_eks_cluster is true and eks_enable_cloudfront_nlb_ingress is false." } output "cloudfront_distribution_domain_name" { - value = !var.use_deployment_mode_external_eks ? module.ingress[0].cloudfront_distribution_domain_name : null - description = "The domain name of the cloudfront distribution" + value = !local.use_eks ? module.ingress[0].cloudfront_distribution_domain_name : (var.create_eks_cluster ? module.eks_cluster[0].cloudfront_distribution_domain_name : null) + description = "The domain name of the CloudFront distribution. Null when create_eks_cluster is true and eks_enable_cloudfront_nlb_ingress is false." } output "cloudfront_distribution_arn" { - value = !var.use_deployment_mode_external_eks ? module.ingress[0].cloudfront_distribution_arn : null + value = !local.use_eks ? module.ingress[0].cloudfront_distribution_arn : (var.create_eks_cluster ? module.eks_cluster[0].cloudfront_distribution_arn : null) description = "The ARN of the cloudfront distribution" } output "cloudfront_distribution_hosted_zone_id" { - value = !var.use_deployment_mode_external_eks ? module.ingress[0].cloudfront_distribution_hosted_zone_id : null - description = "The hosted zone ID of the cloudfront distribution" + value = !local.use_eks ? module.ingress[0].cloudfront_distribution_hosted_zone_id : (var.create_eks_cluster ? module.eks_cluster[0].cloudfront_distribution_hosted_zone_id : null) + description = "The hosted zone ID of the CloudFront distribution. Null when create_eks_cluster is true and eks_enable_cloudfront_nlb_ingress is false." } output "kms_key_arn" { @@ -177,3 +228,80 @@ output "quarantine_lambda_security_group_id" { value = module.services_common.quarantine_lambda_security_group_id description = "ID of the security group for quarantine Lambda functions" } + +output "eks_cluster_name" { + value = var.create_eks_cluster ? module.eks_cluster[0].cluster_name : null + description = "Name of the EKS cluster created by this module. Null when create_eks_cluster is false." +} + +output "eks_cluster_security_group_id" { + value = var.create_eks_cluster ? module.eks_cluster[0].cluster_security_group_id : null + description = "Security group ID attached to the EKS cluster nodes. Null when create_eks_cluster is false." +} + +output "eks_cluster_arn" { + value = var.create_eks_cluster ? module.eks_cluster[0].cluster_arn : null + description = "ARN of the EKS cluster created by this module. Null when create_eks_cluster is false." +} + +output "eks_cluster_endpoint" { + value = var.create_eks_cluster ? module.eks_cluster[0].cluster_endpoint : null + description = "API server endpoint of the EKS cluster. Null when create_eks_cluster is false." +} + +output "eks_cluster_oidc_issuer_url" { + value = var.create_eks_cluster ? module.eks_cluster[0].cluster_oidc_issuer_url : null + description = "OIDC issuer URL of the EKS cluster, used for IRSA. Null when create_eks_cluster is false." +} + +output "eks_cluster_certificate_authority_data" { + value = var.create_eks_cluster ? module.eks_cluster[0].cluster_certificate_authority_data : null + description = "Base64-encoded certificate authority data for the EKS cluster." + sensitive = true +} + +output "eks_node_group_iam_role_arn" { + value = var.create_eks_cluster ? module.eks_cluster[0].node_group_iam_role_arn : null + description = "IAM role ARN shared by all EKS node groups. Null when create_eks_cluster is false." +} + +output "eks_node_group_iam_role_name" { + value = var.create_eks_cluster ? module.eks_cluster[0].node_group_iam_role_name : null + description = "IAM role name shared by all EKS node groups. Null when create_eks_cluster is false." +} + +output "eks_braintrust_api_role_arn" { + value = var.create_eks_cluster ? module.services_common.api_handler_role_arn : null + description = "IAM role ARN for the Braintrust API Pod Identity association." +} + +output "eks_brainstore_role_arn" { + value = var.create_eks_cluster ? module.services_common.brainstore_iam_role_arn : null + description = "IAM role ARN for the Brainstore Pod Identity association." +} + +output "eks_nlb_arn" { + value = var.create_eks_cluster ? module.eks_cluster[0].nlb_arn : null + description = "ARN of the internal NLB used by the EKS API service." +} + +output "eks_nlb_name" { + value = var.create_eks_cluster ? module.eks_cluster[0].nlb_name : null + description = "Name of the internal NLB adopted by the Braintrust API Kubernetes service." +} + +output "eks_nlb_security_group_id" { + value = var.create_eks_cluster ? module.eks_cluster[0].nlb_security_group_id : null + description = "Security group ID attached to the internal NLB." +} + +output "eks_namespace" { + value = var.create_eks_cluster ? local.eks_namespace_resolved : null + description = "Kubernetes namespace where Braintrust workloads are deployed." +} + +output "function_tools_secret_key" { + value = module.services_common.function_tools_secret_key + description = "Secret key used by Braintrust application components for function tools authentication." + sensitive = true +} diff --git a/variables.tf b/variables.tf index 4835c581..1135d1be 100644 --- a/variables.tf +++ b/variables.tf @@ -4,6 +4,8 @@ locals { private_subnet_2_az = var.private_subnet_2_az != null ? var.private_subnet_2_az : data.aws_availability_zones.available.names[1] private_subnet_3_az = var.private_subnet_3_az != null ? var.private_subnet_3_az : data.aws_availability_zones.available.names[2] public_subnet_1_az = var.public_subnet_1_az != null ? var.public_subnet_1_az : data.aws_availability_zones.available.names[0] + public_subnet_2_az = var.public_subnet_2_az != null ? var.public_subnet_2_az : data.aws_availability_zones.available.names[1] + public_subnet_3_az = var.public_subnet_3_az != null ? var.public_subnet_3_az : data.aws_availability_zones.available.names[2] # Lookup and choose an AZ if not provided for Quarantine VPC quarantine_private_subnet_1_az = var.quarantine_private_subnet_1_az != null ? var.quarantine_private_subnet_1_az : data.aws_availability_zones.available.names[0] @@ -121,6 +123,12 @@ variable "existing_public_subnet_1_id" { } } +variable "existing_public_subnet_2_id" { + type = string + default = null + description = "ID of existing public subnet 2 in a second AZ (optional when create_vpc is false; required for ALB which needs subnets in at least two AZs)" +} + variable "private_subnet_1_az" { type = string default = null @@ -139,12 +147,34 @@ variable "private_subnet_3_az" { description = "Availability zone for the third private subnet. Leave blank to choose the third available zone" } +variable "public_subnet_count" { + type = number + default = 3 + description = "Number of public subnets to create when create_vpc is true. Use 1 for NLB-only deployments, 2 or 3 for ALB (which requires subnets in at least two AZs)." + validation { + condition = contains([1, 2, 3], var.public_subnet_count) + error_message = "public_subnet_count must be 1, 2, or 3." + } +} + variable "public_subnet_1_az" { type = string default = null description = "Availability zone for the public subnet. Leave blank to choose the first available zone" } +variable "public_subnet_2_az" { + type = string + default = null + description = "Availability zone for the second public subnet. Leave blank to choose the second available zone." +} + +variable "public_subnet_3_az" { + type = string + default = null + description = "Availability zone for the third public subnet. Leave blank to choose the third available zone." +} + variable "enable_quarantine_vpc" { type = bool description = "Enable the Quarantine VPC to run user defined functions in an isolated environment. If disabled, user defined functions will not be available." @@ -457,13 +487,17 @@ variable "gateway_braintrust_app_url" { } variable "braintrust_api_url" { - description = "Optional. Braintrust API URL used by the gateway when using external EKS deployment mode." + description = "Optional. Braintrust API URL used by the gateway when using external EKS deployment mode, or when create_eks_cluster is true but the module-managed EKS CloudFront/NLB ingress is disabled." type = string default = null validation { - condition = !(var.use_deployment_mode_external_eks && var.enable_llm_gateway) || var.braintrust_api_url != null - error_message = "braintrust_api_url is required when use_deployment_mode_external_eks and enable_llm_gateway are both true." + condition = !(var.enable_llm_gateway && (( + (var.use_deployment_mode_eks || var.use_deployment_mode_external_eks) && !var.create_eks_cluster + ) || ( + var.create_eks_cluster && !var.eks_enable_cloudfront_nlb_ingress + ))) || var.braintrust_api_url != null + error_message = "braintrust_api_url is required when enable_llm_gateway = true and the API URL is not created by this module, including external EKS deployments and create_eks_cluster deployments with eks_enable_cloudfront_nlb_ingress = false." } } @@ -549,13 +583,13 @@ variable "outbound_rate_limit_window_minutes" { } variable "custom_domain" { - description = "Custom domain name for the CloudFront distribution" + description = "Optional custom domain name for the CloudFront distribution. Leave null to use the generated CloudFront domain." type = string default = null } variable "custom_certificate_arn" { - description = "ARN of the ACM certificate for the custom domain" + description = "ARN of the us-east-1 ACM certificate for the custom domain. Leave null when using the generated CloudFront domain." type = string default = null } @@ -647,6 +681,10 @@ variable "brainstore_license_key" { type = string description = "The license key for the Brainstore instance" default = null + validation { + condition = !var.create_eks_cluster || (var.brainstore_license_key != null && trimspace(var.brainstore_license_key) != "") + error_message = "brainstore_license_key is required when create_eks_cluster is true." + } } variable "brainstore_version_override" { @@ -824,7 +862,13 @@ variable "use_global_ai_proxy" { } variable "use_deployment_mode_external_eks" { - description = "Enable EKS deployment mode. When true, disables lambdas, ec2, and ingress submodules. It assumes an EKS deployment is being done outside of terraform." + description = "Deprecated alias for use_deployment_mode_eks. Preserved for backwards compatibility." + type = bool + default = false +} + +variable "use_deployment_mode_eks" { + description = "Enable the EKS/Kubernetes deployment mode. When true, disables lambdas, ec2, and ingress submodules. Use create_eks_cluster to choose whether the EKS cluster is created by this module or managed externally." type = bool default = false } @@ -882,3 +926,344 @@ variable "override_brainstore_iam_role_trust_policy" { description = "Advanced: If provided, this will completely replace the trust policy for the Brainstore IAM role. Must be a valid JSON string representing the IAM trust policy document." default = null } + +#---------------------------------------------------------------------------------------------- +# EKS Cluster (create_eks_cluster = true) +#---------------------------------------------------------------------------------------------- + +variable "create_eks_cluster" { + description = "When true, creates an EKS cluster and Pod Identity IAM roles within this module. Automatically disables Lambda services, EC2 Brainstore, and the CloudFront/API Gateway ingress. For externally-managed clusters, leave false and enable use_deployment_mode_eks instead." + type = bool + default = false + validation { + condition = !var.create_eks_cluster || var.use_deployment_mode_eks || var.use_deployment_mode_external_eks + error_message = "create_eks_cluster requires use_deployment_mode_eks = true (or the deprecated use_deployment_mode_external_eks = true)." + } +} + +variable "eks_use_auto_mode" { + description = "Enable EKS Auto Mode. When true, EKS manages compute (system and services nodes), load balancing, and block storage automatically. System and services managed node groups are not created — EKS uses built-in node pools instead. Brainstore node groups are always created as managed node groups because they require a custom NVMe launch template. Defaults to true." + type = bool + default = true +} + +variable "kubernetes_version" { + description = "Kubernetes version for the EKS cluster. Only used when create_eks_cluster is true." + type = string + default = "1.31" +} + +variable "eks_enable_private_access" { + description = "Whether the EKS private API server endpoint is enabled." + type = bool + default = true +} + +variable "eks_enable_public_access" { + description = "Whether the EKS public API server endpoint is enabled." + type = bool + default = true +} + +variable "eks_public_access_cidrs" { + description = "CIDR blocks allowed to reach the EKS public API server endpoint." + type = list(string) + default = ["0.0.0.0/0"] +} + +variable "eks_access_entries" { + description = "Additional EKS access entries to create when create_eks_cluster is true. Use this to grant explicit human or CI access to the cluster without relying only on the cluster creator identity." + type = map(object({ + principal_arn = string + type = optional(string, "STANDARD") + kubernetes_groups = optional(list(string)) + user_name = optional(string) + policy_associations = optional(map(object({ + policy_arn = string + access_scope = object({ + type = string + namespaces = optional(list(string), []) + }) + })), {}) + })) + default = {} + + validation { + condition = alltrue(flatten([ + for _, entry in var.eks_access_entries : [ + for _, policy_association in entry.policy_associations : ( + contains(["cluster", "namespace"], policy_association.access_scope.type) && + ( + ( + policy_association.access_scope.type == "cluster" && + length(policy_association.access_scope.namespaces) == 0 + ) || + ( + policy_association.access_scope.type == "namespace" && + length(policy_association.access_scope.namespaces) > 0 + ) + ) + ) + ] + ])) + error_message = "Each EKS access policy association must use access_scope.type of cluster with no namespaces, or namespace with at least one namespace." + } +} + +variable "eks_enable_cloudfront_nlb_ingress" { + description = "When true, creates the module-managed CloudFront + private NLB ingress stack for create_eks_cluster deployments. Set false if you want the EKS cluster without the bundled ingress so you can bring your own ingress/controller setup." + type = bool + default = true +} + +variable "eks_cluster_log_types" { + description = "EKS control plane log types to enable." + type = list(string) + default = ["api", "audit", "authenticator"] +} + +variable "eks_cluster_log_retention_days" { + description = "Days to retain EKS control plane logs in CloudWatch." + type = number + default = 90 +} + +variable "eks_enable_node_ssm" { + description = "Enable AWS Systems Manager Session Manager on EKS nodes for debugging." + type = bool + default = false +} + +variable "eks_api_service_account_name" { + description = "Kubernetes service account name for the Braintrust API pods." + type = string + default = "braintrust-api" +} + +variable "eks_brainstore_service_account_name" { + description = "Kubernetes service account name for Brainstore pods." + type = string + default = "brainstore" +} + +variable "eks_aws_load_balancer_controller_service_account" { + description = "Kubernetes service account name for the AWS Load Balancer Controller." + type = string + default = "aws-load-balancer-controller" +} + +variable "eks_node_group_ami_type" { + description = "AMI type for the system and services node groups." + type = string + default = "AL2023_x86_64_STANDARD" +} + +variable "eks_brainstore_node_group_ami_type" { + description = "AMI type for the brainstore node groups. Defaults to Graviton (ARM64) because c8gd NVMe instances are recommended for Brainstore." + type = string + default = "AL2023_ARM_64_STANDARD" +} + +variable "eks_system_node_group_instance_type" { + description = "Instance type for the EKS system node group." + type = string + default = "t3.medium" +} + +variable "eks_system_node_group_desired_size" { + description = "Desired node count for the EKS system node group." + type = number + default = 2 +} + +variable "eks_system_node_group_min_size" { + description = "Minimum node count for the EKS system node group." + type = number + default = 2 +} + +variable "eks_system_node_group_max_size" { + description = "Maximum node count for the EKS system node group." + type = number + default = 4 +} + +variable "eks_system_node_group_disk_size" { + description = "Root EBS volume size in GB for EKS system nodes." + type = number + default = 50 +} + +variable "eks_services_node_group_instance_type" { + description = "Instance type for the EKS services node group." + type = string + default = "r6i.2xlarge" +} + +variable "eks_services_node_group_desired_size" { + description = "Desired node count for the EKS services node group." + type = number + default = 2 +} + +variable "eks_services_node_group_min_size" { + description = "Minimum node count for the EKS services node group." + type = number + default = 2 +} + +variable "eks_services_node_group_max_size" { + description = "Maximum node count for the EKS services node group." + type = number + default = 10 +} + +variable "eks_services_node_group_disk_size" { + description = "Root EBS volume size in GB for EKS services nodes." + type = number + default = 100 +} + +variable "eks_brainstore_reader_node_group_instance_type" { + description = "Instance type for the EKS brainstore reader node group. Must have local NVMe storage." + type = string + default = "c8gd.8xlarge" +} + +variable "eks_brainstore_reader_node_group_desired_size" { + description = "Desired node count for the EKS brainstore reader node group." + type = number + default = 2 +} + +variable "eks_brainstore_reader_node_group_min_size" { + description = "Minimum node count for the EKS brainstore reader node group." + type = number + default = 2 +} + +variable "eks_brainstore_reader_node_group_max_size" { + description = "Maximum node count for the EKS brainstore reader node group." + type = number + default = 10 +} + +variable "eks_brainstore_writer_node_group_instance_type" { + description = "Instance type for the EKS brainstore writer node group. Must have local NVMe storage. Defaults to c8gd.16xlarge (2× the vCPU/memory of the reader default)." + type = string + default = "c8gd.16xlarge" +} + +variable "eks_brainstore_writer_node_group_desired_size" { + description = "Desired node count for the EKS brainstore writer node group." + type = number + default = 1 +} + +variable "eks_brainstore_writer_node_group_min_size" { + description = "Minimum node count for the EKS brainstore writer node group." + type = number + default = 1 +} + +variable "eks_brainstore_writer_node_group_max_size" { + description = "Maximum node count for the EKS brainstore writer node group." + type = number + default = 5 +} + +variable "eks_enable_services_spot_node_group" { + description = "Add a spot node group for services burst capacity alongside the on-demand baseline." + type = bool + default = false +} + +variable "eks_services_spot_node_group_instance_types" { + description = "Instance types for the services spot node group." + type = list(string) + default = ["r6i.2xlarge", "r5.2xlarge", "r6a.2xlarge"] +} + +variable "eks_services_spot_node_group_min_size" { + description = "Minimum node count for the services spot node group." + type = number + default = 0 +} + +variable "eks_services_spot_node_group_max_size" { + description = "Maximum node count for the services spot node group." + type = number + default = 9 +} + +variable "eks_enable_brainstore_spot_node_group" { + description = "Add a spot node group for brainstore burst capacity. Spot interruptions cause cache loss; pods rehydrate from S3 on restart." + type = bool + default = false +} + +variable "eks_brainstore_spot_node_group_instance_types" { + description = "Instance types for the brainstore spot node group. Must be NVMe-backed Graviton instances." + type = list(string) + default = ["c8gd.8xlarge", "c8gd.12xlarge", "c6gd.8xlarge"] +} + +variable "eks_brainstore_spot_node_group_min_size" { + description = "Minimum node count for the brainstore spot node group." + type = number + default = 0 +} + +variable "eks_brainstore_spot_node_group_max_size" { + description = "Maximum node count for the brainstore spot node group." + type = number + default = 9 +} + +variable "eks_enable_brainstore_writer_spot_node_group" { + description = "Add a spot node group for brainstore writer burst capacity. Spot interruptions cause cache loss; pods rehydrate from S3 on restart." + type = bool + default = false +} + +variable "eks_brainstore_writer_spot_node_group_instance_types" { + description = "Instance types for the brainstore writer spot node group. Must be NVMe-backed Graviton instances with 2× the vCPU/memory of reader instances." + type = list(string) + default = ["c8gd.16xlarge", "c7gd.16xlarge", "c6gd.16xlarge"] +} + +variable "eks_brainstore_writer_spot_node_group_min_size" { + description = "Minimum node count for the brainstore writer spot node group." + type = number + default = 0 +} + +variable "eks_brainstore_writer_spot_node_group_max_size" { + description = "Maximum node count for the brainstore writer spot node group." + type = number + default = 9 +} + +variable "eks_vpc_cni_addon_version" { + description = "Version of the VPC CNI addon. Null uses the default for the cluster version." + type = string + default = null +} + +variable "eks_kube_proxy_addon_version" { + description = "Version of the kube-proxy addon. Null uses the default for the cluster version." + type = string + default = null +} + +variable "eks_coredns_addon_version" { + description = "Version of the CoreDNS addon. Null uses the default for the cluster version." + type = string + default = null +} + +variable "eks_pod_identity_addon_version" { + description = "Version of the EKS Pod Identity agent addon. Null uses the default for the cluster version." + type = string + default = null +}