feat(agents): sandbox-bound CLI agents (Codex, Claude Code, custom) with capture + in-box SWE-bench grading by Glorf · Pull Request #1647 · NVIDIA-NeMo/Gym

Glorf · 2026-06-22T13:05:56Z

Summary

A reusable layer for running CLI coding agents inside a Gym sandbox and collecting RL-ready trajectories, on the sandbox-provider + adapter framework. Benchmark-agnostic — only the task metadata and the grader change.

SandboxCliAgent owns the per-rollout lifecycle: start a sandbox -> per-rollout capture proxy -> install + run the CLI in-box -> collect the git patch -> assemble the trajectory -> grade. A new agent is a ~30-line subclass or a YAML manifest.
Agents: codex_swe_agent (Responses wire), claude_code_swe_agent (Messages wire via the translate_anthropic interceptor -> any OpenAI-compatible backend), custom_agent (manifest-driven).
Adapters are sandbox-bound (per rollout): each rollout spins its own capture proxy keyed to a session_id; the in-box *_BASE_URL points at it; torn down on exit. CaptureStore writes durable per-session JSONL (token-ids on a Gym model server). choose_trajectory prefers a well-formed (paired tool calls) trajectory, so both codex and Claude Code produce healthy trajectories.
In-box grading (the only benchmark-specific seam): SWE-bench via the official swebench harness; Terminal-Bench / Harbor by staging the task's tests, running test.sh, and reading the verifier reward file; lightweight pytest membership fallback.

How to run

Per agent — start the agent server, then drive a dataset (swap codex_swe_agent for claude_code_swe_agent / custom_agent):

ng_run "+config_paths=[responses_api_agents/codex_swe_agent/configs/codex_swe_agent.yaml]" \
  +policy_base_url=https://<endpoint> ++codex_swe_agent.responses_api_agents.codex_swe_agent.model=<model-id>
# NEMO_GYM_MODEL_API_KEY = the real upstream key; the in-box agent only ever sees a dummy
# (the per-rollout capture proxy injects the real key on the way upstream).

ng_collect_rollouts +agent_name=codex_swe_agent \
  +input_jsonl_fpath=<dataset>.jsonl +output_jsonl_fpath=rollouts.jsonl +limit=N

Per task — only responses_create_params.metadata differs:

SWE-bench — metadata.instance_dict (repo, version, base_commit, test_patch, FAIL_TO_PASS, PASS_TO_PASS); image resolves __->_1776_; graded by the swebench harness. Example: responses_api_agents/codex_swe_agent/data/example.jsonl.
Terminal-Bench / Harbor — metadata.docker_image (public, auto-mirrored to ECR) + metadata.harbor_tests ({"/tests/test.sh": "...", "/tests/test_outputs.py": "..."}); graded by running test.sh and reading the verifier reward file. Example: responses_api_agents/codex_swe_agent/data/terminal_bench_example.jsonl.

ng_collect_rollouts +agent_name=codex_swe_agent \
  +input_jsonl_fpath=responses_api_agents/codex_swe_agent/data/terminal_bench_example.jsonl \
  +output_jsonl_fpath=tb.jsonl +limit=1

Build full datasets: SWE-bench via HF princeton-nlp/SWE-bench_Verified; Terminal-Bench via harbor datasets download terminal-bench@2.0 (each task's instruction.md + task.toml docker_image + tests/). See the agent READMEs for the full per-agent/per-task matrix.

Validated on real ECS Fargate

SWE-bench Verified: golden patch -> 1.0, no patch -> 0.0; Codex + Claude Code each solve real instances; 10 diverse repos via ng_run + ng_collect_rollouts.
Terminal-Bench 2.0 (Harbor): 5 tasks x Codex + Claude Code — multiple solves (reward 1.0 via the verifier reward file), the grader discriminates, and trajectories are healthy (paired tool calls) for both agents.

Dependencies (stacked PRs)

Stacked on the open base PRs below — its diff includes their changes until they land on main, so review/merge after them. Opened as a draft for that reason.

Sandbox

Sandbox provider framework — NVIDIA-NeMo/Gym#1377
ECS Fargate sandbox provider — hemildesai/Gym#2 (Glorf:feat/ecs-fargate-sandbox)

Adapters / interceptors

Adapter middleware framework (base) — NVIDIA-NeMo/Gym#1518 (Glorf:feat/adapter-base)

Related interceptor families on the same feat/adapter-base (siblings, not required here): caching Glorf/Gym#2, observability Glorf/Gym#3, request-rewriting Glorf/Gym#4.

Test plan

uv run pytest tests/unit_tests/test_sandbox_cli_agent.py tests/unit_tests/test_adapter_capture.py tests/unit_tests/test_adapter_translate_anthropic.py responses_api_agents/codex_swe_agent/tests responses_api_agents/claude_code_swe_agent/tests responses_api_agents/custom_agent/tests
(needs AWS + model creds) graded SWE-bench + Terminal-Bench rollouts on ECS Fargate per the run commands above

Recreated with an upstream head so it can join the GitHub stack (supersedes #1611, which had a fork head).

copy-pr-bot · 2026-06-22T13:06:00Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…ith capture + SWE-bench/Terminal-Bench grading Adds a reusable layer for running CLI coding agents *inside* a Gym sandbox and collecting RL-ready trajectories, built on the sandbox provider + adapter framework. Benchmark-agnostic: only task metadata + the grader change. SandboxCliAgent (nemo_gym/sandbox_cli_agent.py) - Owns the whole per-rollout lifecycle: start a sandbox, stand up a per-rollout capture proxy, install + run the CLI in-box, collect the git patch, assemble the trajectory, and grade. Subclasses supply only a thin seam (build_launch + parse_stdout); adding an agent is a small subclass or YAML. Agents - codex_swe_agent: Codex (Responses wire) as a thin subclass. - claude_code_swe_agent: Claude Code (Messages wire) via the translate_anthropic interceptor so it runs against any OpenAI-compatible backend; reuses claude_code_agent's stream-json parser. - custom_agent: manifest-driven (YAML-only) onboarding. Adapters (sandbox-bound, not model-bound) - capture interceptor + CaptureStore: durable, session-keyed JSONL of model exchanges (token-ids when the backend is a Gym model server); trajectory assembly for chat + responses wires. choose_trajectory prefers a well-formed (paired tool calls) trajectory, so codex's streamed-response capture and claude's duplicate-call redundancy both fall back to the CLI's clean stdout. - translate_anthropic interceptor: Anthropic Messages <-> OpenAI Chat. In-box grading (the only benchmark-specific seam) - SWE-bench: the official swebench harness (make_test_spec eval_script + get_eval_report), like mini_swe_agent_2. - Terminal-Bench / Harbor: per-task public docker_image (auto_mirrored), stage the task's tests, run test.sh, read the verifier reward file. - Fallback: a lightweight pytest membership grader for a custom eval_command. Runnable under ng_run / ng_collect_rollouts; per-agent + per-task examples and a Terminal-Bench example row are in the agent READMEs/data. Agents install nemo-gym[sandbox,sandbox-ecs] (editable) + swebench. Validated on real ECS Fargate: - SWE-bench Verified: golden -> 1.0, no-patch -> 0.0; Codex + Claude Code each solve real instances; 10 diverse repos via ng_run + ng_collect_rollouts. - Terminal-Bench 2.0 (Harbor): across 5 tasks x Codex + Claude Code — multiple solves (reward 1.0 via the verifier reward file), grader discriminates, and trajectories are healthy (paired tool calls) for both agents. Unit-tested: grader, image-tag mapping, SWE-bench instance reconstruction, Harbor grading, trajectory selection, capture assembly, Anthropic<->OpenAI translation. Note: stacked on the sandbox provider PR (feat/ecs-fargate-sandbox) and the adapter framework PR (feat/adapter-base); review/merge after them. Signed-off-by: Michal Bien <mbien@nvidia.com>

cmunley1 · 2026-06-23T18:30:19Z

+    return fallback, False
+
+
+def swebench_image_tag(instance_id: str) -> str:


dont think we should have swebench specific things in nemogym core.

cmunley1 · 2026-06-23T18:33:07Z

+    ) -> SandboxCliAgentVerifyResponse:
+        params = body.responses_create_params
+        metadata = self._task_metadata(body)
+        # Per-task workdir: SWE-bench images use /testbed (the config default); Harbor/TB


is this specific to swe / tb and not generic?

cmunley1 · 2026-06-23T18:33:49Z

+  responses_api_agents:
+    claude_code_swe_agent:
+      entrypoint: app.py
+      # Model ROOT (no /v1); Claude's /v1/messages is translated to /v1/chat/completions.


this should go through antropic_model_server right?

cmunley1 · 2026-06-23T18:34:42Z

+        "stream-json",
+        "--verbose",
+        "--dangerously-skip-permissions",
+        "--bare",


this has drifted from claude code implementation on main which parameterizes bare etc, also why its important to reuse rather than rewrite agents across environments.

cmunley1 · 2026-06-23T18:35:32Z

@@ -0,0 +1 @@
+{"instance_id": "astropy__astropy-12907", "responses_create_params": {"input": "Fix the bug described below in this repository. Make the minimal code change so the project's tests pass. Do not modify test files.\n\nModeling's `separability_matrix` does not compute separability correctly for nested CompoundModels\nConsider the following model:\r\n\r\n```python\r\nfrom astropy.modeling import models as m\r\nfrom astropy.modeling.separable import separability_matrix\r\n\r\ncm = m.Linear1D(10) & m.Linear1D(5)\r\n```\r\n\r\nIt's separability matrix as you might expect is a diagonal:\r\n\r\n```python\r\n>>> separability_matrix(cm)\r\narray([[ True, False],\r\n       [False,  True]])\r\n```\r\n\r\nIf I make the model more complex:\r\n```python\r\n>>> separability_matrix(m.Pix2Sky_TAN() & m.Linear1D(10) & m.Linear1D(5))\r\narray([[ True,  True, False, False],\r\n       [ True,  True, False, False],\r\n       [False, False,  True, False],\r\n       [False, False, False,  True]])\r\n```\r\n\r\nThe output matrix is again, as expected, the outputs and inputs to the linear models are separable and independent of each other.\r\n\r\nIf however, I nest these compound models:\r\n```python\r\n>>> separability_matrix(m.Pix2Sky_TAN() & cm)\r\narray([[ True,  True, False, False],\r\n       [ True,  True, False, False],\r\n       [False, False,  True,  True],\r\n       [False, False,  True,  True]])\r\n```\r\nSuddenly the inputs and outputs are no longer separable?\r\n\r\nThis feels like a bug to me, but I might be missing something?\n", "metadata": {"instance_id": "astropy__astropy-12907", "instance_dict": "{\"instance_id\": \"astropy__astropy-12907\", \"repo\": \"astropy/astropy\", \"version\": \"4.3\", \"base_commit\": \"d16bfe05a744909de4b27f5875fe0d4ed41ce607\", \"environment_setup_commit\": \"298ccb478e6bf092953bca67a3d29dc6c35f6752\", \"patch\": \"diff --git a/astropy/modeling/separable.py b/astropy/modeling/separable.py\\n--- a/astropy/modeling/separable.py\\n+++ b/astropy/modeling/separable.py\\n@@ -242,7 +242,7 @@ def _cstack(left, right):\\n         cright = _coord_matrix(right, 'right', noutp)\\n     else:\\n         cright = np.zeros((noutp, right.shape[1]))\\n-        cright[-right.shape[0]:, -right.shape[1]:] = 1\\n+        cright[-right.shape[0]:, -right.shape[1]:] = right\\n \\n     return np.hstack([cleft, cright])\\n \\n\", \"test_patch\": \"diff --git a/astropy/modeling/tests/test_separable.py b/astropy/modeling/tests/test_separable.py\\n--- a/astropy/modeling/tests/test_separable.py\\n+++ b/astropy/modeling/tests/test_separable.py\\n@@ -28,6 +28,13 @@\\n p1 = models.Polynomial1D(1, name='p1')\\n \\n \\n+cm_4d_expected = (np.array([False, False, True, True]),\\n+                  np.array([[True,  True,  False, False],\\n+                            [True,  True,  False, False],\\n+                            [False, False, True,  False],\\n+                            [False, False, False, True]]))\\n+\\n+\\n compound_models = {\\n     'cm1': (map3 & sh1 | rot & sh1 | sh1 & sh2 & sh1,\\n             (np.array([False, False, True]),\\n@@ -52,7 +59,17 @@\\n     'cm7': (map2 | p2 & sh1,\\n             (np.array([False, True]),\\n              np.array([[True, False], [False, True]]))\\n-            )\\n+            ),\\n+    'cm8': (rot & (sh1 & sh2), cm_4d_expected),\\n+    'cm9': (rot & sh1 & sh2, cm_4d_expected),\\n+    'cm10': ((rot & sh1) & sh2, cm_4d_expected),\\n+    'cm11': (rot & sh1 & (scl1 & scl2),\\n+             (np.array([False, False, True, True, True]),\\n+              np.array([[True,  True,  False, False, False],\\n+                        [True,  True,  False, False, False],\\n+                        [False, False, True,  False, False],\\n+                        [False, False, False, True,  False],\\n+                        [False, False, False, False, True]]))),\\n }\\n \\n \\n\", \"problem_statement\": \"Modeling's `separability_matrix` does not compute separability correctly for nested CompoundModels\\nConsider the following model:\\r\\n\\r\\n```python\\r\\nfrom astropy.modeling import models as m\\r\\nfrom astropy.modeling.separable import separability_matrix\\r\\n\\r\\ncm = m.Linear1D(10) & m.Linear1D(5)\\r\\n```\\r\\n\\r\\nIt's separability matrix as you might expect is a diagonal:\\r\\n\\r\\n```python\\r\\n>>> separability_matrix(cm)\\r\\narray([[ True, False],\\r\\n       [False,  True]])\\r\\n```\\r\\n\\r\\nIf I make the model more complex:\\r\\n```python\\r\\n>>> separability_matrix(m.Pix2Sky_TAN() & m.Linear1D(10) & m.Linear1D(5))\\r\\narray([[ True,  True, False, False],\\r\\n       [ True,  True, False, False],\\r\\n       [False, False,  True, False],\\r\\n       [False, False, False,  True]])\\r\\n```\\r\\n\\r\\nThe output matrix is again, as expected, the outputs and inputs to the linear models are separable and independent of each other.\\r\\n\\r\\nIf however, I nest these compound models:\\r\\n```python\\r\\n>>> separability_matrix(m.Pix2Sky_TAN() & cm)\\r\\narray([[ True,  True, False, False],\\r\\n       [ True,  True, False, False],\\r\\n       [False, False,  True,  True],\\r\\n       [False, False,  True,  True]])\\r\\n```\\r\\nSuddenly the inputs and outputs are no longer separable?\\r\\n\\r\\nThis feels like a bug to me, but I might be missing something?\\n\", \"hints_text\": \"\", \"created_at\": \"2022-03-03T15:14:54Z\", \"FAIL_TO_PASS\": [\"astropy/modeling/tests/test_separable.py::test_separable[compound_model6-result6]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model9-result9]\"], \"PASS_TO_PASS\": [\"astropy/modeling/tests/test_separable.py::test_coord_matrix\", \"astropy/modeling/tests/test_separable.py::test_cdot\", \"astropy/modeling/tests/test_separable.py::test_cstack\", \"astropy/modeling/tests/test_separable.py::test_arith_oper\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model0-result0]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model1-result1]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model2-result2]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model3-result3]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model4-result4]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model5-result5]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model7-result7]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model8-result8]\", \"astropy/modeling/tests/test_separable.py::test_custom_model_separable\"]}"}, "max_output_tokens": 16384}}


should add 5 examples + example rollouts and metrics

cmunley1

can you help find places to dedup work among
#1572
#1631
#1677

I think a key thing we need among these implementations is complete reusability of harnesses like claude code and codex among all current and future environments such as swe, terminal, math, search, all benchmarks, etc.

cmunley1 · 2026-06-23T18:39:58Z

have you validated training and logprob mismatch is in acceptable range? how do you determine which messages are subagent or main agent or otherwise non monotonic?

cmunley1 · 2026-06-23T18:52:11Z

+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Anthropic Messages <-> OpenAI Chat Completions translation.


what about anthropic_model_server that is on another PR

ECS Fargate `SandboxProvider` on top of the sandbox API (#1377, now merged to `main`). Auto-mirrors public images to ECR on demand; SSH reverse-tunnel for exec / file-transfer / model egress. Rebased onto `main` now that the sandbox base (#1377) has landed — this PR stands on its own and is ready for review. **Sandbox-bound agents line of work:** **ECS Fargate (this PR)** → adapter middleware base (#1646) → sandbox-bound CLI agents + capture (#1647). Interceptor follow-ups on the adapter base: #1649 (caching), #1650 (observability), #1651 (rewrites). 3 commits, +5974: ECS provider + engine, on-demand ECR mirroring + conda activation fix, and 233 unit tests (engine.py to 99%). `uv.lock` reconciled with main's security upgrades (#1657). --------- Signed-off-by: Michal Bien <mbien@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Glorf force-pushed the feat/adapter-base branch from 35a1b91 to 8e7dee2 Compare June 22, 2026 14:19

Glorf force-pushed the feat/sandbox-cli-agents branch 2 times, most recently from ef55b6c to bb43704 Compare June 22, 2026 15:58

Glorf mentioned this pull request Jun 23, 2026

feat(sandbox): ECS Fargate sandbox provider #1645

Merged

Glorf force-pushed the feat/adapter-base branch from 8e7dee2 to 4916143 Compare June 23, 2026 09:06

Glorf requested a review from a team as a code owner June 23, 2026 09:06

Glorf force-pushed the feat/sandbox-cli-agents branch 2 times, most recently from 4c03181 to 86e53b4 Compare June 23, 2026 10:57

Glorf force-pushed the feat/adapter-base branch from 2399ce2 to 45c8b8c Compare June 23, 2026 11:54

Glorf force-pushed the feat/sandbox-cli-agents branch 2 times, most recently from a25995b to 358934f Compare June 23, 2026 12:23

Glorf force-pushed the feat/adapter-base branch from 45c8b8c to 73051c7 Compare June 23, 2026 12:29

Glorf force-pushed the feat/sandbox-cli-agents branch from 358934f to fdaeae2 Compare June 23, 2026 12:30

Glorf force-pushed the feat/adapter-base branch from 73051c7 to c4313fb Compare June 23, 2026 12:38

Glorf force-pushed the feat/sandbox-cli-agents branch from fdaeae2 to 64a140e Compare June 23, 2026 12:38

cmunley1 reviewed Jun 23, 2026

View reviewed changes

cmunley1 requested changes Jun 23, 2026

View reviewed changes

cmunley1 reviewed Jun 23, 2026

View reviewed changes

Glorf marked this pull request as draft June 24, 2026 08:01

Glorf closed this Jun 24, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(agents): sandbox-bound CLI agents (Codex, Claude Code, custom) with capture + in-box SWE-bench grading#1647

feat(agents): sandbox-bound CLI agents (Codex, Claude Code, custom) with capture + in-box SWE-bench grading#1647
Glorf wants to merge 1 commit into
feat/adapter-basefrom
feat/sandbox-cli-agents

Glorf commented Jun 22, 2026

Uh oh!

copy-pr-bot Bot commented Jun 22, 2026

Uh oh!

cmunley1 Jun 23, 2026

Uh oh!

cmunley1 Jun 23, 2026

Uh oh!

cmunley1 Jun 23, 2026

Uh oh!

cmunley1 Jun 23, 2026

Uh oh!

cmunley1 Jun 23, 2026

Uh oh!

cmunley1 left a comment

Uh oh!

cmunley1 commented Jun 23, 2026

Uh oh!

cmunley1 Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return fallback, False


		def swebench_image_tag(instance_id: str) -> str:

		@@ -0,0 +1 @@
		{"instance_id": "astropy__astropy-12907", "responses_create_params": {"input": "Fix the bug described below in this repository. Make the minimal code change so the project's tests pass. Do not modify test files.\n\nModeling's `separability_matrix` does not compute separability correctly for nested CompoundModels\nConsider the following model:\r\n\r\n```python\r\nfrom astropy.modeling import models as m\r\nfrom astropy.modeling.separable import separability_matrix\r\n\r\ncm = m.Linear1D(10) & m.Linear1D(5)\r\n```\r\n\r\nIt's separability matrix as you might expect is a diagonal:\r\n\r\n```python\r\n>>> separability_matrix(cm)\r\narray([[ True, False],\r\n [False, True]])\r\n```\r\n\r\nIf I make the model more complex:\r\n```python\r\n>>> separability_matrix(m.Pix2Sky_TAN() & m.Linear1D(10) & m.Linear1D(5))\r\narray([[ True, True, False, False],\r\n [ True, True, False, False],\r\n [False, False, True, False],\r\n [False, False, False, True]])\r\n```\r\n\r\nThe output matrix is again, as expected, the outputs and inputs to the linear models are separable and independent of each other.\r\n\r\nIf however, I nest these compound models:\r\n```python\r\n>>> separability_matrix(m.Pix2Sky_TAN() & cm)\r\narray([[ True, True, False, False],\r\n [ True, True, False, False],\r\n [False, False, True, True],\r\n [False, False, True, True]])\r\n```\r\nSuddenly the inputs and outputs are no longer separable?\r\n\r\nThis feels like a bug to me, but I might be missing something?\n", "metadata": {"instance_id": "astropy__astropy-12907", "instance_dict": "{\"instance_id\": \"astropy__astropy-12907\", \"repo\": \"astropy/astropy\", \"version\": \"4.3\", \"base_commit\": \"d16bfe05a744909de4b27f5875fe0d4ed41ce607\", \"environment_setup_commit\": \"298ccb478e6bf092953bca67a3d29dc6c35f6752\", \"patch\": \"diff --git a/astropy/modeling/separable.py b/astropy/modeling/separable.py\\n--- a/astropy/modeling/separable.py\\n+++ b/astropy/modeling/separable.py\\n@@ -242,7 +242,7 @@ def _cstack(left, right):\\n cright = _coord_matrix(right, 'right', noutp)\\n else:\\n cright = np.zeros((noutp, right.shape[1]))\\n- cright[-right.shape[0]:, -right.shape[1]:] = 1\\n+ cright[-right.shape[0]:, -right.shape[1]:] = right\\n \\n return np.hstack([cleft, cright])\\n \\n\", \"test_patch\": \"diff --git a/astropy/modeling/tests/test_separable.py b/astropy/modeling/tests/test_separable.py\\n--- a/astropy/modeling/tests/test_separable.py\\n+++ b/astropy/modeling/tests/test_separable.py\\n@@ -28,6 +28,13 @@\\n p1 = models.Polynomial1D(1, name='p1')\\n \\n \\n+cm_4d_expected = (np.array([False, False, True, True]),\\n+ np.array([[True, True, False, False],\\n+ [True, True, False, False],\\n+ [False, False, True, False],\\n+ [False, False, False, True]]))\\n+\\n+\\n compound_models = {\\n 'cm1': (map3 & sh1 \| rot & sh1 \| sh1 & sh2 & sh1,\\n (np.array([False, False, True]),\\n@@ -52,7 +59,17 @@\\n 'cm7': (map2 \| p2 & sh1,\\n (np.array([False, True]),\\n np.array([[True, False], [False, True]]))\\n- )\\n+ ),\\n+ 'cm8': (rot & (sh1 & sh2), cm_4d_expected),\\n+ 'cm9': (rot & sh1 & sh2, cm_4d_expected),\\n+ 'cm10': ((rot & sh1) & sh2, cm_4d_expected),\\n+ 'cm11': (rot & sh1 & (scl1 & scl2),\\n+ (np.array([False, False, True, True, True]),\\n+ np.array([[True, True, False, False, False],\\n+ [True, True, False, False, False],\\n+ [False, False, True, False, False],\\n+ [False, False, False, True, False],\\n+ [False, False, False, False, True]]))),\\n }\\n \\n \\n\", \"problem_statement\": \"Modeling's `separability_matrix` does not compute separability correctly for nested CompoundModels\\nConsider the following model:\\r\\n\\r\\n```python\\r\\nfrom astropy.modeling import models as m\\r\\nfrom astropy.modeling.separable import separability_matrix\\r\\n\\r\\ncm = m.Linear1D(10) & m.Linear1D(5)\\r\\n```\\r\\n\\r\\nIt's separability matrix as you might expect is a diagonal:\\r\\n\\r\\n```python\\r\\n>>> separability_matrix(cm)\\r\\narray([[ True, False],\\r\\n [False, True]])\\r\\n```\\r\\n\\r\\nIf I make the model more complex:\\r\\n```python\\r\\n>>> separability_matrix(m.Pix2Sky_TAN() & m.Linear1D(10) & m.Linear1D(5))\\r\\narray([[ True, True, False, False],\\r\\n [ True, True, False, False],\\r\\n [False, False, True, False],\\r\\n [False, False, False, True]])\\r\\n```\\r\\n\\r\\nThe output matrix is again, as expected, the outputs and inputs to the linear models are separable and independent of each other.\\r\\n\\r\\nIf however, I nest these compound models:\\r\\n```python\\r\\n>>> separability_matrix(m.Pix2Sky_TAN() & cm)\\r\\narray([[ True, True, False, False],\\r\\n [ True, True, False, False],\\r\\n [False, False, True, True],\\r\\n [False, False, True, True]])\\r\\n```\\r\\nSuddenly the inputs and outputs are no longer separable?\\r\\n\\r\\nThis feels like a bug to me, but I might be missing something?\\n\", \"hints_text\": \"\", \"created_at\": \"2022-03-03T15:14:54Z\", \"FAIL_TO_PASS\": [\"astropy/modeling/tests/test_separable.py::test_separable[compound_model6-result6]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model9-result9]\"], \"PASS_TO_PASS\": [\"astropy/modeling/tests/test_separable.py::test_coord_matrix\", \"astropy/modeling/tests/test_separable.py::test_cdot\", \"astropy/modeling/tests/test_separable.py::test_cstack\", \"astropy/modeling/tests/test_separable.py::test_arith_oper\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model0-result0]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model1-result1]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model2-result2]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model3-result3]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model4-result4]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model5-result5]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model7-result7]\", \"astropy/modeling/tests/test_separable.py::test_separable[compound_model8-result8]\", \"astropy/modeling/tests/test_separable.py::test_custom_model_separable\"]}"}, "max_output_tokens": 16384}}

Uh oh!

Conversation

Glorf commented Jun 22, 2026

Summary

How to run

Validated on real ECS Fargate

Dependencies (stacked PRs)

Test plan

Uh oh!

copy-pr-bot Bot commented Jun 22, 2026

Uh oh!

cmunley1 Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

cmunley1 Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

cmunley1 Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

cmunley1 Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

cmunley1 Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

cmunley1 left a comment

Choose a reason for hiding this comment

Uh oh!

cmunley1 commented Jun 23, 2026

Uh oh!

cmunley1 Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants