|
| 1 | +--- |
| 2 | +title: "ECS Fargate" |
| 3 | +description: "Run sandboxes as AWS ECS Fargate tasks reached over an SSH sidecar." |
| 4 | +position: 3 |
| 5 | +--- |
| 6 | + |
| 7 | +The `ecs_fargate` provider runs each sandbox as an AWS ECS Fargate task reached over an SSH sidecar. It implements the provider-neutral `SandboxProvider` contract, so any sandbox-backed agent or resources server can use it by selecting `ecs_fargate` in its sandbox config. |
| 8 | + |
| 9 | +## Setup |
| 10 | + |
| 11 | +ECS Fargate needs the AWS SDK and a provisioned account/region: |
| 12 | + |
| 13 | +```bash |
| 14 | +uv pip install boto3 |
| 15 | +``` |
| 16 | + |
| 17 | +- **Infrastructure.** The reference Terraform stack publishes its outputs to SSM at `/<ssm_project>/ecs-sandbox/config` (`ssm_project` defaults to `harbor`): cluster, subnets, security groups, task/execution roles, ECR mirror, EFS, and the SSH-sidecar key ARNs. A missing parameter raises an actionable error. |
| 18 | +- **Credentials.** `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (or an instance role) plus `AWS_REGION`. |
| 19 | +- **Network.** The host must reach each task's SSH sidecar port (`52222`). Run inside the sandbox VPC/peered network, or allow the host IP on the sidecar security group. Exec, file transfer, and model reverse-tunnels all ride this SSH connection. |
| 20 | + |
| 21 | +## Provider Configuration |
| 22 | + |
| 23 | +Region-only is enough; everything else auto-discovers from SSM (explicit YAML always wins): |
| 24 | + |
| 25 | +```yaml |
| 26 | +sandbox_provider: |
| 27 | + ecs_fargate: |
| 28 | + region: ${oc.env:AWS_REGION} |
| 29 | + cpu: "2048" |
| 30 | + memory: "8192" |
| 31 | + ephemeral_storage_gib: 50 |
| 32 | +``` |
| 33 | +
|
| 34 | +| Field | Default | Purpose | |
| 35 | +| --- | --- | --- | |
| 36 | +| `region` | — | AWS region; enables SSM auto-discovery when `cluster` is omitted | |
| 37 | +| `cpu` / `memory` | `"4096"` / `"8192"` | Fargate task size (CPU units / MiB) when set directly | |
| 38 | +| `ephemeral_storage_gib` | 20 (implicit) | Task scratch disk; explicit values must be 21–200 | |
| 39 | +| `auto_mirror` | `true` | Mirror a missing public image into the ECR mirror on demand | |
| 40 | +| `ssm_project` | `harbor` | SSM namespace for auto-discovery | |
| 41 | +| `environment_dir` | — | Build the task image from a Dockerfile directory via CodeBuild instead of using a prebuilt image | |
| 42 | + |
| 43 | +Cluster, subnets, security groups, roles, ECR mirror, EFS defaults, and the SSH-sidecar key ARNs are filled in from SSM when omitted. |
| 44 | + |
| 45 | +## Provider Options |
| 46 | + |
| 47 | +Per-sandbox options go in `SandboxSpec.provider_options`: |
| 48 | + |
| 49 | +| Key | Purpose | |
| 50 | +| --- | --- | |
| 51 | +| `volumes` | EFS mounts: a list of `{"container_path": "/mnt/efs", "efs": true}` entries. Each inherits the provider's `efs_filesystem_id` / `efs_access_point_id` unless it names its own. | |
| 52 | +| `outside_endpoints` | Host URLs exposed inside the sandbox via an SSH reverse tunnel, as `{"url": ..., "env_var": ...}` (e.g. a model server). | |
| 53 | +| `environment_dir` | Dockerfile directory to build the task image from via CodeBuild. | |
| 54 | + |
| 55 | +Common `SandboxSpec` fields map onto the task as follows: `ttl_s` → sidecar watchdog that stops the task, `ready_timeout_s` → task-startup timeout, `env` / `files` / `workdir` → container environment, seed files, and working directory. |
| 56 | + |
| 57 | +## Resource Mapping and Isolation |
| 58 | + |
| 59 | +`SandboxResources` maps onto the Fargate task definition: |
| 60 | + |
| 61 | +| `SandboxResources` | Fargate | |
| 62 | +| --- | --- | |
| 63 | +| `cpu` (vCPU) | task CPU units (`cpu * 1024`), validated against Fargate's CPU/memory pairs | |
| 64 | +| `memory_mib` | task memory (MiB) | |
| 65 | +| `disk_gib` | task ephemeral storage (21–200 GiB) | |
| 66 | +| `gpu` | unsupported — raises `SandboxCreateError` | |
| 67 | + |
| 68 | +Each sandbox is a dedicated Fargate task with its own kernel, network namespace, and microVM boundary — there is no host sharing between sandboxes. The orchestrator holds a per-task SSH connection for exec and file transfer, and TTL is enforced by a sidecar watchdog that stops the task. |
| 69 | + |
| 70 | +## Images and On-Demand Mirroring |
| 71 | + |
| 72 | +ECS pulls task images from the account ECR mirror rather than their origin registry. A bare/public image (e.g. `docker.io/swebench/sweb.eval.x86_64.<id>:latest`) resolves to the mirror tag `<ecr_repository>:<sanitized-name>`. Resolution order: |
| 73 | + |
| 74 | +1. `environment_dir` set → build the image via CodeBuild and use it. |
| 75 | +2. Image is already an ECR reference → use verbatim (never re-mirrored). |
| 76 | +3. Bare/public name + `auto_mirror=true` → mirror into ECR on demand (CodeBuild pull → retag → push) during `create`, then launch. |
| 77 | + |
| 78 | +The first task for a new image waits on a one-time build (~1–3 min for typical SWE-bench images); later tasks hit the ECR cache, and concurrent tasks for the same image de-duplicate onto a single build. Set `auto_mirror: false` to require a pre-populated mirror and fail fast on a miss. |
| 79 | + |
| 80 | +## First-Run Example |
| 81 | + |
| 82 | +Drive a sandbox directly through the provider-neutral API: |
| 83 | + |
| 84 | +```python |
| 85 | +import asyncio |
| 86 | +
|
| 87 | +from nemo_gym.sandbox import AsyncSandbox, SandboxResources, SandboxSpec |
| 88 | +
|
| 89 | +
|
| 90 | +provider_config = {"ecs_fargate": {"region": "us-east-1"}} |
| 91 | +spec = SandboxSpec( |
| 92 | + image="python:3.12-slim", |
| 93 | + ttl_s=1800, |
| 94 | + ready_timeout_s=300, |
| 95 | + resources=SandboxResources(cpu=2, memory_mib=8192), |
| 96 | +) |
| 97 | +
|
| 98 | +
|
| 99 | +async def main() -> None: |
| 100 | + async with AsyncSandbox(provider_config, spec) as sandbox: |
| 101 | + await sandbox.start() |
| 102 | + result = await sandbox.exec("python3 --version", timeout_s=60) |
| 103 | + print(result.stdout or result.stderr) |
| 104 | +
|
| 105 | +
|
| 106 | +asyncio.run(main()) |
| 107 | +``` |
| 108 | + |
| 109 | +For an end-to-end agent run, `mini_swe_agent_2` ships a ready config at `responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_ecs_fargate.yaml` that selects this provider with region-only auto-discovery. |
| 110 | + |
| 111 | +## Troubleshooting |
| 112 | + |
| 113 | +| Symptom | Cause / fix | |
| 114 | +| --- | --- | |
| 115 | +| `SSM parameter '/…/ecs-sandbox/config' not found` | Infrastructure is not provisioned in that region. Run the Terraform stack, or set the ECS fields explicitly in YAML. | |
| 116 | +| Exec/SSH times out after the task reaches RUNNING | The host cannot reach the sidecar port `52222`. Run inside the sandbox VPC or allow the host IP on the sidecar security group. | |
| 117 | +| `Fargate memory for cpu=… must be …` | The CPU/memory pair is not a supported Fargate combination. Pick a valid pair. | |
| 118 | +| `ephemeral storage must be between 21 and 200 GiB` | `disk_gib` is out of range. Omit it for the implicit 20 GiB default. | |
| 119 | +| First task is slow or the image pull fails | The first task mirrors the image via CodeBuild (~1–3 min); later tasks hit the ECR cache. Set `auto_mirror: false` to require a pre-staged mirror. | |
| 120 | +| S3 staging objects accumulate | The orchestrator role lacks `s3:DeleteObject`. Grant it, or set a bucket lifecycle policy to reap `*/ecs-sandbox/*` staging artifacts. | |
| 121 | +| `ECS Fargate does not support GPU sandboxes` | Fargate has no GPU; use a GPU-capable provider instead. | |
| 122 | + |
| 123 | +<Warning> |
| 124 | +The reference sidecar security group allows `0.0.0.0/0` on `52222` for convenience. Restrict it to the orchestrator's egress IP (or move to a private/peered path) before non-smoke use. |
| 125 | +</Warning> |
0 commit comments