Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@ This is a Terraform module that deploys the Braintrust hybrid data plane on AWS.
│ ├── braintrust-data-plane/ # Production example
│ ├── braintrust-data-plane-sandbox/ # Sandbox/testing example
│ ├── braintrust-data-plane-external-eks-quarantine/
│ └── cloudfront-logging/
│ ├── cloudfront-logging/
│ ├── eks-auto-mode/ # EKS Auto Mode example with Terraform-managed Helm deployment
│ └── eks-terraform-managed/ # EKS example with Terraform-managed node groups and manual Helm install
├── scripts/ # Operational helper scripts
└── mise.toml # Tool versions and tasks (terraform, tflint, uv)
```
Expand Down
48 changes: 48 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,54 @@ If you're using a brand new AWS account for your Braintrust data plane you will
## Module Configuration
All module input variables and outputs are documented inline in the module's Terraform files (see `variables.tf`, `outputs.tf`, and the submodules for details).

## EKS Deployment Mode

Use `use_deployment_mode_eks = true` together with `create_eks_cluster = true` to have this module build the AWS and EKS infrastructure for a Braintrust deployment. In this mode, the root module stops at the infrastructure boundary: it creates the VPC-facing resources, database, Redis, storage, IAM, EKS cluster, node capacity, and the outputs needed by a Kubernetes deployment layer. The bundled CloudFront + private NLB ingress path is enabled by default and can be disabled with `eks_enable_cloudfront_nlb_ingress = false` if you want to bring your own ingress.

The two EKS examples show the intended compositions on top of that base:

- [`examples/eks-auto-mode`](examples/eks-auto-mode) is the one-click path. It composes the root module with `modules/eks-deploy` to create the cluster, bundled CloudFront/NLB ingress, and deploy the Braintrust app in the same apply, using EKS Auto Mode for compute.
- [`examples/eks-terraform-managed`](examples/eks-terraform-managed) is the handoff path. It creates the AWS data plane and a Terraform-managed EKS cluster with explicit node groups, disables the bundled ingress by default, and leaves the final Braintrust chart install plus ingress choice to the customer.

The older `use_deployment_mode_external_eks` variable remains supported as a deprecated alias for backwards compatibility.

The module enables the private EKS API endpoint by default. If you keep the
public endpoint enabled, set `eks_public_access_cidrs` to explicit operator or
CI egress CIDRs for production instead of leaving it open to `0.0.0.0/0`. The
open default is allowed for compatibility and bootstrap convenience. If you
disable public access, run Terraform, `kubectl`, and Helm steps that contact the
cluster from the VPC or a connected network.

Use `eks_access_entries` to grant long-lived human or CI access to module-managed
clusters. The cluster creator receives bootstrap admin access, but additional
operators should be represented explicitly with EKS access entries and scoped
access policies.

If your EKS cluster is managed outside this module and you need a module-managed
quarantine VPC for user-defined functions, use the external EKS quarantine
example:

- [`examples/braintrust-data-plane-external-eks-quarantine`](examples/braintrust-data-plane-external-eks-quarantine) for EKS with a quarantine VPC for user-defined functions

By default, EKS deployments with `eks_enable_cloudfront_nlb_ingress = true` use the generated CloudFront `*.cloudfront.net` hostname and the default CloudFront TLS certificate. Leave `custom_domain = null` and `custom_certificate_arn = null` for this baseline.

To use a customer-owned hostname with the bundled ingress path, set `custom_domain` and provide an ACM certificate ARN from `us-east-1` through `custom_certificate_arn`. CloudFront requires viewer certificates for custom aliases to live in `us-east-1`, even when the data plane runs in another AWS region.

The module does not create or validate the certificate for you, and it does not create DNS records automatically. The expected workflow is:

1. Request or import an ACM certificate for the exact hostname you want CloudFront to serve, in `us-east-1`.
2. Validate that certificate in ACM.
3. Set `custom_domain` and `custom_certificate_arn` on the module.
4. Apply Terraform so the CloudFront distribution is updated with the alias and viewer certificate.
5. Create a DNS alias record for `custom_domain` that points at the module outputs `cloudfront_distribution_domain_name` and `cloudfront_distribution_hosted_zone_id`.
6. Register the resulting `api_url` output in the Braintrust dashboard.

If you leave both values null, the module continues to use the generated CloudFront hostname and default certificate. If you set `custom_domain`, you should also set `custom_certificate_arn`; CloudFront will reject a custom alias without a matching `us-east-1` ACM certificate.

If you set `eks_enable_cloudfront_nlb_ingress = false`, the module does not create the private NLB, CloudFront distribution, or related CloudFront outputs. In that mode, `api_url` is also null until you provide your own ingress path.

`waf_acl_id` is optional in both the Lambda/API Gateway and EKS/CloudFront paths. If set, the module attaches your existing WAF Web ACL to the CloudFront distribution; if not set, no WAF is attached.

## Useful scripts

### dump-logs.sh
Expand Down
2 changes: 2 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ Example configurations for deploying the Braintrust data plane on AWS. Copy the
| [braintrust-data-plane](./braintrust-data-plane/) | Standard production-sized deployment. Start here for most use cases. |
| [braintrust-data-plane-sandbox](./braintrust-data-plane-sandbox/) | Minimal deployment for infrastructure testing. Downsized instances, no quarantine VPC, deletion protection disabled. Not for workload testing. |
| [braintrust-data-plane-external-eks-quarantine](./braintrust-data-plane-external-eks-quarantine/) | Deployment with an external EKS cluster and quarantine VPC for user-defined functions. |
| [eks-terraform-managed](./eks-terraform-managed/) | Terraform-managed node-group cluster example. Terraform creates the AWS data plane and EKS cluster, disables the bundled CloudFront/NLB ingress by default, and leaves the app install and ingress choice to the customer. |
| [eks-auto-mode](./eks-auto-mode/) | Full one-click Auto Mode example. Terraform creates the AWS data plane, EKS cluster, bundled CloudFront/NLB ingress, Kubernetes bootstrap resources, and the Braintrust application deployment. |
| [cloudfront-logging](./cloudfront-logging/) | Add-on example for enabling CloudFront access logging to S3. |

See the [self-hosting documentation](https://www.braintrust.dev/docs/guides/self-hosting/aws) for full setup instructions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

This is an example configuration for deploying a Braintrust data plane with the following characteristics:

- **External EKS Deployment Mode**: Uses `use_deployment_mode_external_eks = true` to deploy services on an existing EKS cluster managed outside of Terraform
- **External EKS Deployment Mode**: Uses `use_deployment_mode_eks = true` to deploy services on an existing EKS cluster managed outside of Terraform
- **EKS Pod Identity**: Enables `enable_eks_pod_identity = true` for the Braintrust IAM roles
- **Quarantine VPC**: Deploys the Quarantine VPC (`enable_quarantine_vpc = true`) for running user-defined functions in an isolated environment

## Quarantine VPC IAM Permissions

The Quarantine VPC IAM permissions are automatically deployed when `enable_quarantine_vpc = true`, even when using `use_deployment_mode_external_eks = true`. These IAM resources are created in the `services-common` module:
The Quarantine VPC IAM permissions are automatically deployed when `enable_quarantine_vpc = true`, even when using `use_deployment_mode_eks = true`. These IAM resources are created in the `services-common` module:

- `QuarantineInvokeRole` - Role used by the API handler to invoke quarantined functions
- `QuarantineFunctionRole` - Role used by the quarantined Lambda functions
Expand All @@ -21,7 +21,7 @@ These resources are available for use by your EKS-deployed services via EKS Pod
### Required Settings

1. **EKS Configuration**:
- Set `use_deployment_mode_external_eks = true`
- Set `use_deployment_mode_eks = true`
- Set `enable_eks_pod_identity = true`

2. **Quarantine VPC**:
Expand Down Expand Up @@ -123,3 +123,4 @@ Ensure these don't conflict with:

- See the main [README.md](../../README.md) for general information
- See [examples/braintrust-data-plane/README.md](../braintrust-data-plane/README.md) for standard deployment examples
- See [helm/braintrust/examples/aws-eks-quarantine/values.yaml](../../../helm/braintrust/examples/aws-eks-quarantine/values.yaml) for a matching Helm values example
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,13 @@
# with Quarantine VPC enabled and EKS Pod Identity.
#
# This configuration:
# - Uses external EKS deployment mode (use_deployment_mode_external_eks = true)
# - Uses external EKS deployment mode (use_deployment_mode_eks = true)
# - Enables EKS Pod Identity (enable_eks_pod_identity = true)
# - Deploys the Quarantine VPC (enable_quarantine_vpc = true)
# - Deploys all IAM permissions needed for the Quarantine VPC

module "braintrust-data-plane" {
# Using local source for testing - change to GitHub source for production
source = "../../"
# source = "github.com/braintrustdata/terraform-braintrust-data-plane"
source = "github.com/braintrustdata/terraform-braintrust-data-plane"
# Append '?ref=<version_tag>' to lock to a specific version of the module.

# This is primarily used for tagging and naming resources in your AWS account.
Expand All @@ -28,7 +26,7 @@ module "braintrust-data-plane" {
# Enable external EKS deployment mode
# When true, disables lambdas, ec2, and ingress submodules.
# It assumes an EKS deployment is being done outside of terraform.
use_deployment_mode_external_eks = true
use_deployment_mode_eks = true

# With external EKS, there are additional configurations that must be applied after the EKS cluster has been created outside of this module.
# Enable EKS Pod Identity for the Braintrust IAM roles
Expand Down
161 changes: 161 additions & 0 deletions examples/eks-auto-mode/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# Braintrust Data Plane On EKS Auto Mode

This is the full one-click EKS Auto Mode example.

Terraform creates:

- the Braintrust AWS data plane resources
- the EKS cluster
- the CloudFront edge and private NLB
- the Kubernetes bootstrap resources
- the AWS Load Balancer Controller
- the Brainstore Auto Mode NodeClass and NodePool resources
- the Braintrust Helm release

If you want Terraform-managed node groups but a manual app install, use
[`../eks-terraform-managed/`](../eks-terraform-managed/).

## Required Inputs

- `braintrust_org_name`
- `brainstore_license_key`
- `eks_helm_chart_version`

## EKS API Endpoint Access

This example manages Kubernetes and Helm resources during the same Terraform
apply that creates the EKS cluster. If you disable the public EKS API endpoint,
the Terraform runner must be inside the VPC or on a connected network that can
reach the private endpoint, such as VPN, Direct Connect, Transit Gateway, a
VPC-attached CI runner, or an SSM-reached admin host.

For a laptop or public CI bootstrap, keep private access enabled. The example
defaults `eks_public_access_cidrs` to `["0.0.0.0/0"]` for compatibility, but
production deployments should restrict the public endpoint to explicit operator
or CI egress CIDRs:

```hcl
eks_enable_public_access = true
eks_public_access_cidrs = ["203.0.113.10/32"]
```

For private-only production applies, run Terraform from a VPC-connected
environment and set:

```hcl
eks_enable_public_access = false
```

## EKS Access Entries

The cluster creator keeps bootstrap admin access. Add explicit long-lived human
or CI access through `eks_access_entries`; for example:

```hcl
eks_access_entries = {
platform_admin = {
principal_arn = "arn:aws:iam::123456789012:role/PlatformAdmin"
policy_associations = {
cluster_admin = {
policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
access_scope = {
type = "cluster"
}
}
}
}

support_viewer = {
principal_arn = "arn:aws:iam::123456789012:role/BraintrustSupportViewer"
policy_associations = {
braintrust_logs = {
policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
access_scope = {
type = "namespace"
namespaces = ["braintrust"]
}
}
}
}

ci_deployer = {
principal_arn = "arn:aws:iam::123456789012:role/BraintrustDeploy"
policy_associations = {
braintrust_deploy = {
policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy"
access_scope = {
type = "namespace"
namespaces = ["braintrust"]
}
}
}
}
}
```

Adjust `namespaces` if you change `eks_namespace`.

## Apply

```bash
terraform init
terraform apply
```

The example sets conservative Brainstore reader / fast-reader / writer defaults so
the dedicated Auto Mode NodePools have allocatable headroom on their fixed
`4xlarge` / `8xlarge` instance sizes. Override `eks_brainstore_reader_instance_sizes`
or `eks_brainstore_writer_instance_sizes` if you need larger reader or writer nodes.
If you increase Brainstore replicas or
requests, wait for the dedicated Brainstore pods to become `Running` before
registering the API URL.

Check rollout status first:

```bash
$(terraform output -raw connect_to_cluster)
kubectl get pods -n "$(terraform output -raw eks_namespace)" -w
```

After the API and Brainstore pods are ready, register the API URL in the Braintrust dashboard:

```bash
terraform output -raw api_url
```

This example leaves `eks_enable_cloudfront_nlb_ingress = true`, so Terraform
also provisions the module-managed CloudFront + private NLB ingress path.

For production public deployments, create an AWS WAF Web ACL with the managed
rules and rate limits that match your environment, then attach it to the
module-managed CloudFront distribution:

```hcl
waf_acl_id = "arn:aws:wafv2:us-east-1:123456789012:global/webacl/braintrust/..."
```

If you need Kubernetes NetworkPolicy enforcement, add the VPC CNI network policy
configuration and NetworkPolicy manifests in your environment. This example
does not configure those policies for you.

## Custom Domain

To use your own hostname, provide:

- `custom_domain`
- `custom_certificate_arn`

The ACM certificate must live in `us-east-1`. After apply, create a DNS alias
to:

```bash
terraform output -raw cloudfront_distribution_domain_name
terraform output -raw cloudfront_distribution_hosted_zone_id
```

## Connect To The Cluster

```bash
$(terraform output -raw connect_to_cluster)
kubectl get pods -n "$(terraform output -raw eks_namespace)"
```
Loading
Loading