feat(sandbox): ECS Fargate sandbox provider by Glorf · Pull Request #1645 · NVIDIA-NeMo/Gym

Glorf · 2026-06-22T13:00:23Z

ECS Fargate SandboxProvider on top of the sandbox API (#1377, now merged to main). Auto-mirrors public images to ECR on demand; SSH reverse-tunnel for exec / file-transfer / model egress.

Rebased onto main now that the sandbox base (#1377) has landed — this PR stands on its own and is ready for review.

Sandbox-bound agents line of work:
ECS Fargate (this PR) → adapter middleware base (#1646) → sandbox-bound CLI agents + capture (#1647). Interceptor follow-ups on the adapter base: #1649 (caching), #1650 (observability), #1651 (rewrites).

3 commits, +5974: ECS provider + engine, on-demand ECR mirroring + conda activation fix, and 233 unit tests (engine.py to 99%). uv.lock reconciled with main's security upgrades (#1657).

copy-pr-bot · 2026-06-22T13:00:27Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

hemildesai

2 small comments

@hemildesai

Per @hemildesai's review: - mini_swe_agent_2: reword the conda-activation comment to be provider-agnostic (the sandbox exec shell is non-login across apptainer/docker/ECS); the command itself was already generic, so no behavior change. - pyproject: fold boto3 into the `sandbox` extra and drop the separate `sandbox-ecs` extra; update the mini_swe_agent_2 requirement, test comments, and uv.lock to match. Signed-off-by: Michal Bien <mbien@nvidia.com>

ananthsub

added a bunch of comments inline. the most critical are that the sandbox interface/API was updated from when the initial fargate smoke test was run, so the provider configs + implementation needs to be updated accordingly

@hemildesai

Per @hemildesai's review: - mini_swe_agent_2: reword the conda-activation comment to be provider-agnostic (the sandbox exec shell is non-login across apptainer/docker/ECS); the command itself was already generic, so no behavior change. - pyproject: fold boto3 into the `sandbox` extra and drop the separate `sandbox-ecs` extra; update the mini_swe_agent_2 requirement, test comments, and uv.lock to match. Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf · 2026-06-26T10:39:10Z

@ananthsub @hemildesai thanks for your comments! They were adressed, could you give it another try? :)

github-actions · 2026-06-26T11:55:05Z

🌿 Preview your docs: https://nvidia-preview-feat-ecs-fargate-sandbox.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

Port NEL's ECS Fargate sandboxing into Gym as a third sandbox provider, stacked on the provider framework from #1377. - nemo_gym/sandbox/providers/ecs_fargate/{engine,provider}.py — NEL's engine lifted (task-def registration + SSM caching, RunTask with capacity retries, SSH sidecar + reverse-tunnel outside-endpoint routing, in-container exec server, CodeBuild->ECR image build) behind a thin adapter that keeps per-sandbox state in SandboxHandle.raw. - registry loader + `sandbox-ecs` extra (boto3, lazy-imported). - region-only config via SSM autodiscovery (/<ssm_project>/ecs-sandbox/config), matching NEL. - mini_swe_agent_2 ecs_fargate config + 14 unit tests (AWS/SSH mocked). Fast-follow P0 (separate PR): route the model endpoint over Teleport as an internal option instead of the globally-exposed SSH reverse tunnel that security flagged; SSH tunneling stays available for community use. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Michal Bien <mbien@nvidia.com>

- ECS Fargate provider mirrors a missing public image into the ECR mirror on demand during create (ImageBuilder.ensure_mirrored, gated by the new auto_mirror config, default on), with the same in-flight dedup as the environment_dir build path. Callers no longer pre-stage images. - Fix MiniSWESandboxEnvironment conda activation: source conda from known install roots instead of `conda info --base`, which fails in the ECS exec server's non-login shell and was silently dropping the golden gold-patch apply (zeroing reward). - Add the sandbox-ecs extra to mini_swe_agent_2 requirements so the agent's per-server venv can run the ECS provider under ng_run. - Docs: ECS Fargate provider README (general, provider-scoped) + a concise SWE-bench-on-ECS run guide in the agent README. - Tests for mirror resolution/dedup/auto-mirror hook and conda command build. Signed-off-by: Michal Bien <mbien@nvidia.com>

…CI fixes Add tests/unit_tests/test_ecs_fargate_engine.py (208 boto3/SSH/HTTP-mocked tests asserting real call args, retries, parsing and error paths), raising ecs_fargate/engine.py to 99% and the repo total to 97.4% (>=96 gate). Tag secretsmanager-ARN test fixtures with allowlist-secret pragmas and apply ruff-format. Fixes the Test (coverage), lint and secrets-detector checks. Signed-off-by: Michal Bien <mbien@nvidia.com>

@hemildesai

Per @hemildesai's review: - mini_swe_agent_2: reword the conda-activation comment to be provider-agnostic (the sandbox exec shell is non-login across apptainer/docker/ECS); the command itself was already generic, so no behavior change. - pyproject: fold boto3 into the `sandbox` extra and drop the separate `sandbox-ecs` extra; update the mini_swe_agent_2 requirement, test comments, and uv.lock to match. Signed-off-by: Michal Bien <mbien@nvidia.com>

@ananthsub

…S bugs) P0 fixes from @ananthsub's review: - Sandbox API drift: config sandbox_spec.timeout_s -> ttl_s; drop stale sandbox_environment_kwargs.delete; provider.close() drops the removed `delete` kwarg; provider maps spec.resources (cpu/memory_mib/disk_gib) + spec.ttl_s onto the ECS config (rejects GPU, which Fargate lacks). - Image-build dedup: wait on the in-flight event OUTSIDE cls._lock (one waiter no longer serializes all builds) and verify via ECR on wake, so a failed build no longer returns a URL for an image that was never pushed. - Fargate ephemeral storage: omit the field unless >=21 GiB is requested and validate 21-200 (an explicit 20 is rejected by RegisterTaskDefinition). - Shell quoting: shlex.quote env values + cwd in exec (injection / breakage). - {task_ip}: fail loudly instead of baking a literal placeholder into container env (the IP is only known after the task starts). Tests: rewrite the build/mirror dedup tests to the corrected contract + add failure-path, ephemeral-range, task_ip-guard, and resource/ttl-mapping tests. Signed-off-by: Michal Bien <mbien@nvidia.com>

@ananthsub

…aks) P1 fixes from @ananthsub's review: - Build semaphore: initialize once; never replace a live semaphore when a later caller passes a different build_parallelism (a permit held against the old object would drift the cap). - Fargate cpu/memory: validate the pair (not independent maxes) before RegisterTaskDefinition -- cpu/memory derived as separate max-of-base-and-config could form an unsupported combination. - S3 artifacts: delete the CodeBuild context zip after the build (try/finally) and the per-rollout upload tarballs on cleanup. The process-cached exec-server script is left alone -- it is shared across sandboxes, so per-instance deletion would be unsafe. - _get_task_public_ip: when assign_public_ip is enabled, keep retrying for the PublicIp to propagate before falling back to the private IP on the last attempt (a private IP is unreachable from the orchestrator and otherwise yields a misleading SSH timeout later). Tests: cpu/memory validation, public-IP retry, and best-effort S3 delete. Signed-off-by: Michal Bien <mbien@nvidia.com>

@ananthsub

…, CodeBuild) Final batch from @ananthsub's review: - Mirror injection: validate src_image (dataset-controlled) against a strict registry/repo/tag/ digest charset before it is interpolated into the privileged CodeBuild docker pull/tag commands. - Async hygiene: offload the blocking SSH wait_ready in the exec() reconnect path via asyncio.to_thread so it can't stall the event loop. - EFS: a volume may inherit the provider-level efs_filesystem_id / efs_access_point_id defaults (opt in via `efs: true`); a volume with no resolvable filesystem id fails fast. - RepositoryNotFound: image_exists_in_ecr fails fast with a clear error for a missing ECR repo instead of reporting "image absent" and building/pushing into a non-existent repo. - CodeBuild: reuse a single shared project (created idempotently) instead of one per build, so per-build projects don't accumulate. - _free_port: documented the TOCTOU window (mitigated by SshTunnel.open() retries). Tests: image-ref validation, EFS default inheritance + missing-fs error, RepositoryNotFound fail-fast, shared CodeBuild project; updated dedup/codebuild tests to the new contracts. Signed-off-by: Michal Bien <mbien@nvidia.com>

Signed-off-by: Michal Bien <mbien@nvidia.com>

- Large S3 uploads (>512 KiB) now land at the requested remote path; the tar was keyed by the local temp basename and extracted into the dest dir, so the file previously landed under the wrong name. - Surface S3 staging cleanup gaps once (missing s3:DeleteObject) instead of silently swallowing them; cleanup stays best-effort and never raises. - Strip nemo-evaluator-next/FEP/NEL provenance and condense verbose docstrings and comments to terse one-liners (no logic changes). Signed-off-by: Michal Bien <mbien@nvidia.com>

- Replace the provider README with a Fern page following the sandbox provider-page shape (setup / config / provider_options / resource mapping + isolation / first-run example / troubleshooting). - Add a minimal sandbox/index.mdx so the section renders standalone; complements the sandbox API docs structure without depending on that PR. Signed-off-by: Michal Bien <mbien@nvidia.com>

Rebased onto main, which merged #1708 (decouple sandbox provider config from the agent config). Align the ECS Fargate provider with that pattern: - Add a standalone provider config nemo_gym/sandbox/providers/ecs_fargate/configs/ecs_fargate.yaml -- a top-level `sandbox` block selected via `sandbox_provider: sandbox`, matching opensandbox/apptainer. Region-only; task cpu/memory/disk come from sandbox_spec.resources. - Drop the agent-coupled configs/mini_swe_agent_ecs_fargate.yaml; the generic mini_swe_agent_2.yaml now runs on ECS by adding the provider config path. - Bump the generic config's sandbox_spec.resources.disk_gib 20 -> 30 so it is valid on every provider (Fargate rejects an explicit 20; 21-200 only). - Update the ECS Fargate docs (Fern page + agent README) to the config-paths recipe. - Reconcile uv.lock (boto3 / botocore, sandbox extra) after the rebase. Signed-off-by: Michal Bien <mbien@nvidia.com>

iguehring-nvidia

@Glorf could you add a comment on how this was tested apart from the unit tests? For example, do we have a dummy run with Gym harbor terminal-bench + ecs sandbox? Otherwise, looks good to me. Great work! Thx!

ananthsub · 2026-07-06T15:25:36Z

+AWS_REGION=us-east-1 ng_run "+config_paths=[$CONFIG_PATHS]" \
+    ++mini_swe_agent_2.responses_api_agents.mini_swe_agent_2.run_golden=true
+
+ng_collect_rollouts +agent_name=mini_swe_agent_2 \
+    +input_jsonl_fpath=data/swe_verified_smoke.jsonl \
+    +output_jsonl_fpath=results/ecs_golden.jsonl \
+    +limit=1 +num_repeats=1 +num_samples_in_parallel=1
+```
+
+A resolved instance returns reward `1.0` with `tests_status` populated.
+
+**Real rollout** — drop `run_golden` and point the model server at a live
+OpenAI-compatible endpoint (`policy_model_name` is the model id sent upstream):
+
+```bash
+AWS_REGION=us-east-1 ng_run "+config_paths=[$CONFIG_PATHS]" \
+    ++policy_base_url=https://<endpoint>/v1 ++policy_api_key=<key> ++policy_model_name=<model-id>
+
+ng_collect_rollouts +agent_name=mini_swe_agent_2 \
+    +input_jsonl_fpath=data/swe_verified_smoke.jsonl \
+    +output_jsonl_fpath=results/ecs_rollout.jsonl \
+    +limit=8 +num_repeats=1 +num_samples_in_parallel=8 \
+    '+responses_create_params={max_output_tokens: 16384, temperature: 0.6, top_p: 0.95}'


use the new CLI commands in the documentaiton

Glorf · 2026-07-06T18:48:19Z

@kajalj22 could you help on the automation side? 🙏

Glorf · 2026-07-07T08:57:38Z

@NVIDIA-NeMo/automation can I get your review 🙏

Glorf requested a review from a team as a code owner June 22, 2026 13:00

Glorf mentioned this pull request Jun 22, 2026

feat(sandbox): ECS Fargate sandbox provider hemildesai/Gym#2

Closed

Glorf force-pushed the feat/ecs-fargate-sandbox branch 2 times, most recently from a80772d to 11124f9 Compare June 23, 2026 08:41

Glorf changed the base branch from sandbox-api-part-1 to main June 23, 2026 08:48

Glorf closed this Jun 23, 2026

Glorf reopened this Jun 23, 2026

anwithk requested review from ananthsub and hemildesai June 23, 2026 15:28

cwing-nvidia mentioned this pull request Jun 23, 2026

feat(sandbox): ECS Fargate sandbox provider #1686

Closed

5 tasks

cwing-nvidia linked an issue Jun 23, 2026 that may be closed by this pull request

feat(sandbox): ECS Fargate sandbox provider #1686

Closed

5 tasks

cwing-nvidia mentioned this pull request Jun 23, 2026

epic: built-in sandbox providers #1685

Open

10 tasks

hemildesai reviewed Jun 24, 2026

View reviewed changes

Comment thread responses_api_agents/mini_swe_agent_2/sandbox_environment.py Outdated

Comment thread pyproject.toml Outdated

ananthsub reviewed Jun 24, 2026

View reviewed changes

Comment thread responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_ecs_fargate.yaml Outdated

Comment thread responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_ecs_fargate.yaml Outdated

Comment thread nemo_gym/sandbox/providers/ecs_fargate/engine.py

ananthsub reviewed Jun 24, 2026

View reviewed changes

ananthsub requested changes Jun 24, 2026

View reviewed changes

Comment thread nemo_gym/sandbox/providers/ecs_fargate/engine.py Outdated

Comment thread nemo_gym/sandbox/providers/ecs_fargate/engine.py

Comment thread nemo_gym/sandbox/providers/ecs_fargate/engine.py

cwing-nvidia mentioned this pull request Jun 25, 2026

docs: add Concepts page for sandboxes (runtime vs sandbox vs provider) #1723

Open

Glorf force-pushed the feat/ecs-fargate-sandbox branch from f511533 to 72e0299 Compare June 26, 2026 08:42

Glorf requested review from ananthsub and hemildesai June 26, 2026 10:38

Glorf and others added 5 commits July 6, 2026 10:40

Glorf added 6 commits July 6, 2026 10:42

style(sandbox): ruff-format ECS provider test (line wrap)

ea0a92a

Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf force-pushed the feat/ecs-fargate-sandbox branch from a378c42 to dbeb031 Compare July 6, 2026 09:10

iguehring-nvidia approved these changes Jul 6, 2026

View reviewed changes

ananthsub approved these changes Jul 6, 2026

View reviewed changes

kajalj22 approved these changes Jul 8, 2026

View reviewed changes

Merge branch 'main' into feat/ecs-fargate-sandbox

9628d00

Glorf enabled auto-merge (squash) July 8, 2026 16:13

Glorf merged commit 9b005eb into main Jul 8, 2026
16 checks passed

Glorf deleted the feat/ecs-fargate-sandbox branch July 8, 2026 16:14

Uh oh!

Conversation

Glorf commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Jun 22, 2026

Uh oh!

hemildesai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ananthsub left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Glorf commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iguehring-nvidia left a comment

Choose a reason for hiding this comment

Uh oh!

ananthsub Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

Glorf commented Jul 6, 2026

Uh oh!

Glorf commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Glorf commented Jun 22, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading