feat(cli): config_composer for agent/dataset composition (RFC M3) by wprazuch · Pull Request #1673 · NVIDIA-NeMo/Gym

wprazuch · 2026-06-23T13:43:19Z

Summary

Adds nemo_gym/config_composer.py (RFC M3 / friction 6): a pure, resolution-safe composer that rewrites an already-merged OmegaConf DictConfig (the kind produced by Gym's Hydra stack from a benchmark config.yaml) along two axes — agent and dataset.

It never starts servers, reads secrets, resolves interpolations (resolve=False), or emits Hydra override tokens. It returns a new DictConfig; an empty request is a pure passthrough.

What it does

find_agent_block_key — locate the single top-level block whose responses_api_agents.<type> carries a type: benchmark dataset (mirrors benchmarks.py); raise NoComposableAgentBlockError for zero or ambiguous targets (candidates listed).
substitute_agent — swap the composable harness, loading the new agent via resolve_agent_config_path(new_agent, require_composable=True). A Pattern B (self-contained) agent raises agent_registry.AgentNotComposableError (propagated). The env's resources_server / model_server (name: policy_model) / datasets win over the agent config's own values, and the inner block is re-keyed to <new_agent>.
substitute_dataset_params — edit num_repeats / prompt_config on the single benchmark dataset entry; no-op for None.
substitute_model — documented no-op shim. The model axis is deferred to the CLI's --model* flags, which run before composition.
_validate_no_mandatory_placeholders — reject any remaining ??? (OmegaConf.is_missing) field in the agent block and its referenced resources-server block, without resolving interpolations.
compose — pipeline: find_agent_block_key → substitute_agent (if requested) → substitute_dataset_params → _validate_no_mandatory_placeholders.

Tests

tests/unit_tests/test_config_composer.py — 24 tests, 100% module coverage. Covers dataset-param substitution, agent swap (wiring carry-over + re-keying), placeholder validation (raise on ???, pass when filled), empty-request passthrough, zero/ambiguous agent block → NoComposableAgentBlockError, and Pattern B → AgentNotComposableError via require_composable. Synthetic merged DictConfigs are built inline; the agent resolver is injected as a stub.

copy-pr-bot · 2026-06-23T13:43:23Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…duplicates the asset selector Per a duplication audit against main: resolve_agent_config_path was dead code in this PR (no CLI wires it) and reinvented the unified CLI`s generic name->config resolution (cli/main.py _asset_config_path) — the same pattern #1635 deliberately removed on the environment side. Drop it (and its AgentNotFound/Variant/NotComposable errors and the duplicate _did_you_mean). This PR is now purely discovery + Pattern A/B classification + `gym list agents`. Run-by-name resolution (with the require_composable guard) will land with its real consumer, the config composer (#1673), as an `agent` asset row rather than a parallel resolver. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

@cmunley1

… core) (#1671) ## What Adds `nemo_gym.agent_registry` + the **`gym list agents`** command — the foundation for RFC **M3 / friction 6** ("every agent×environment combination requires a separate config"). Maps an agent's short `<name>` (directory under `responses_api_agents/`) to its config variant(s) and classifies how each harness composes. ## `gym list agents` (CLI) Lists every discovered agent and its composition pattern — the one thing the unified CLI's generic `--<asset>` selectors can't surface. Table + `--json`, mirroring `gym list benchmarks` / `gym list environments` (#1635). ``` $ gym list agents agent composition variants description simple_agent composable (A) simple_agent swe_agents self-contained (B) swebench_openhands, ... Software engineering tasks ... ... ``` Across the repo: **23 agents — 15 Pattern A (composable), 8 Pattern B (self-contained).** ## API - `discover_agents() -> Dict[str, AgentEntry]` — one entry per agent dir; `AgentEntry(name, path, config_paths, self_contained, description)` with a `.variants` map (variant stem → config). - `resolve_agent_config_path(name, variant=None, *, require_composable=False) -> str` — explicit variant wins; a single config is used directly; a variant named after the agent is the default when several exist; otherwise raises. The run-by-name primitive consumed by the composer (#1673). - Errors: `AgentNotFoundError` (did-you-mean), `AgentVariantError` (zero-config / ambiguous), `AgentNotComposableError` (for the composer's `require_composable` guard). ## Composition classification (`self_contained`) - **Pattern A (`self_contained=False`):** references a *separate* `resources_server` → wireable into a compatible environment (e.g. `simple_agent`, `langgraph_agent`). Which harness↔resources-server pairings are actually compatible is the **composer's** concern, not the registry's. - **Pattern B (`self_contained=True`):** bundles its own environment/framework or external LLM loop (`agent_framework`; e.g. `swe_agents`, `harbor_agent`, `verifiers_agent`, `claude_code_agent`) and runs with its own config. `require_composable=True` rejects these with a clear error. - Non-agent YAML in a `configs/` dir is filtered out; zero-config agents (e.g. `aviary_agent`) are still discovered (their wiring lives in a paired benchmark/resources config). Resolution-safe (`resolve=False, throw_on_missing=False`; never starts servers), so it works with unset secrets. > Reflects @cmunley1's review: classification reframed around Pattern A vs self-contained B, field renamed `composable`→`self_contained`, and cross-pattern compatibility documented as the composer's job (separate commit, history preserved). ## Tests `test_agent_registry.py` (21 cases — A/B classification, external-harness carve-out, zero-config, variant selection, did-you-mean, `require_composable`) + `test_cli_agents.py` / `test_cli_main.py` (`gym list agents` table / empty / `--json` / router dispatch). Part of epic #1205 (M3 / friction 6). --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

@cmunley1

… core) (#1671) ## What Adds `nemo_gym.agent_registry` + the **`gym list agents`** command — the foundation for RFC **M3 / friction 6** ("every agent×environment combination requires a separate config"). Maps an agent's short `<name>` (directory under `responses_api_agents/`) to its config variant(s) and classifies how each harness composes. ## `gym list agents` (CLI) Lists every discovered agent and its composition pattern — the one thing the unified CLI's generic `--<asset>` selectors can't surface. Table + `--json`, mirroring `gym list benchmarks` / `gym list environments` (#1635). ``` $ gym list agents agent composition variants description simple_agent composable (A) simple_agent swe_agents self-contained (B) swebench_openhands, ... Software engineering tasks ... ... ``` Across the repo: **23 agents — 15 Pattern A (composable), 8 Pattern B (self-contained).** ## API - `discover_agents() -> Dict[str, AgentEntry]` — one entry per agent dir; `AgentEntry(name, path, config_paths, self_contained, description)` with a `.variants` map (variant stem → config). - `resolve_agent_config_path(name, variant=None, *, require_composable=False) -> str` — explicit variant wins; a single config is used directly; a variant named after the agent is the default when several exist; otherwise raises. The run-by-name primitive consumed by the composer (#1673). - Errors: `AgentNotFoundError` (did-you-mean), `AgentVariantError` (zero-config / ambiguous), `AgentNotComposableError` (for the composer's `require_composable` guard). ## Composition classification (`self_contained`) - **Pattern A (`self_contained=False`):** references a *separate* `resources_server` → wireable into a compatible environment (e.g. `simple_agent`, `langgraph_agent`). Which harness↔resources-server pairings are actually compatible is the **composer's** concern, not the registry's. - **Pattern B (`self_contained=True`):** bundles its own environment/framework or external LLM loop (`agent_framework`; e.g. `swe_agents`, `harbor_agent`, `verifiers_agent`, `claude_code_agent`) and runs with its own config. `require_composable=True` rejects these with a clear error. - Non-agent YAML in a `configs/` dir is filtered out; zero-config agents (e.g. `aviary_agent`) are still discovered (their wiring lives in a paired benchmark/resources config). Resolution-safe (`resolve=False, throw_on_missing=False`; never starts servers), so it works with unset secrets. > Reflects @cmunley1's review: classification reframed around Pattern A vs self-contained B, field renamed `composable`→`self_contained`, and cross-pattern compatibility documented as the composer's job (separate commit, history preserved). ## Tests `test_agent_registry.py` (21 cases — A/B classification, external-harness carve-out, zero-config, variant selection, did-you-mean, `require_composable`) + `test_cli_agents.py` / `test_cli_main.py` (`gym list agents` table / empty / `--json` / router dispatch). Part of epic #1205 (M3 / friction 6). --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

PR #1673 added the `gym env compose` command (agent/dataset composition) but did not document it. Add it to the Environments quick-reference list and a dedicated section with flags and examples, mirroring the sibling commands. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

github-actions · 2026-06-29T12:32:19Z

🌿 Preview your docs: https://nvidia-preview-wprazuch-config-composer.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

nemo/gym/reference/cli-commands

Resolve PR #1673 conflicts with main (now includes #1804/#1805/#1806). The gym env compose work coexists with #1805's install-root _asset_config_path and the agent asset selector in nemo_gym/cli/main.py; both `env compose` and `env validate` remain registered. Auto-merge verified: no conflict markers, 167 unit tests pass (only the pre-existing TestDidYouMean argparse failures), and `gym env compose --benchmark gsm8k --agent simple_agent` + `gym env validate` both work. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ition (RFC M3) Compose a benchmark with a swapped agent harness and/or edited benchmark-dataset params, without writing a bespoke per-combination config (epic #1205, criterion C3): - nemo_gym/config_composer.py: compose(), substitute_agent() (wires the env's resources_server/model_server/datasets into the swapped agent, filling a generic agent's mandatory resources_server.name), find_agent_block_key, substitute_dataset_params, _validate_no_mandatory_placeholders (scans for remaining ??? WITHOUT resolving interpolations), and a composability guard (Pattern B rejected). - gym env compose command + an agent asset selector in the unified CLI. - docs: document gym env compose in the CLI reference. Rebased onto current main (composes with #1805/#1806/#1807): #1807's env.py install-root resolution + validate preserved; gym env compose and gym env validate coexist. Verified: compose --benchmark gsm8k --agent simple_agent wires resources_server; validate works; env-test resolution intact. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

wprazuch · 2026-07-01T16:23:49Z

Closing as deferred

wprazuch force-pushed the wprazuch/agent-registry branch from 707d816 to f6da402 Compare June 24, 2026 07:01

wprazuch requested a review from a team as a code owner June 24, 2026 07:01

wprazuch force-pushed the wprazuch/config-composer branch from 2949bea to 1910101 Compare June 24, 2026 07:05

wprazuch force-pushed the wprazuch/agent-registry branch 2 times, most recently from 9740d8f to d14dc18 Compare June 24, 2026 07:18

wprazuch force-pushed the wprazuch/config-composer branch from 1910101 to 4ea15e4 Compare June 24, 2026 07:20

wprazuch force-pushed the wprazuch/agent-registry branch from d14dc18 to 8c34493 Compare June 24, 2026 07:21

wprazuch force-pushed the wprazuch/config-composer branch 2 times, most recently from 2a9eaf8 to ad4cbf3 Compare June 24, 2026 07:58

wprazuch marked this pull request as draft June 24, 2026 08:02

wprazuch mentioned this pull request Jun 24, 2026

feat: agent registry — name-based agent discovery + composability (M3 core) #1671

Merged

wprazuch force-pushed the wprazuch/agent-registry branch from 21d5994 to 1e62870 Compare June 25, 2026 12:29

Base automatically changed from wprazuch/agent-registry to main June 25, 2026 13:18

wprazuch force-pushed the wprazuch/config-composer branch from ad4cbf3 to 8c4143d Compare June 25, 2026 14:01

wprazuch marked this pull request as ready for review June 25, 2026 14:03

ritaneves mentioned this pull request Jun 25, 2026

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

ritaneves linked an issue Jun 25, 2026 that may be closed by this pull request

6 - Configuration Friction #1736

Open

1 task

wprazuch mentioned this pull request Jun 26, 2026

docs: source: union, clean config errors, and new discovery/validate CLI commands #1754

Merged

wprazuch mentioned this pull request Jun 29, 2026

fix: gym dataset collate creates the artifact dir when missing (run-from-wheel) #1811

Merged

wprazuch force-pushed the wprazuch/config-composer branch from 533ea92 to 38e90c0 Compare June 29, 2026 14:38

copy-pr-bot Bot temporarily deployed to public June 29, 2026 14:39 Inactive

wprazuch force-pushed the wprazuch/config-composer branch from 38e90c0 to 72ef9e6 Compare June 29, 2026 14:39

wprazuch force-pushed the wprazuch/config-composer branch from 72ef9e6 to 1cb55e6 Compare June 29, 2026 15:58

Merge branch 'main' into wprazuch/config-composer

0314f1e

anwithk requested a review from cmunley1 June 30, 2026 02:12

wprazuch mentioned this pull request Jul 1, 2026

docs: use name/flavor selectors instead of internal config paths in data/collate examples #1856

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(cli): config_composer for agent/dataset composition (RFC M3)#1673

feat(cli): config_composer for agent/dataset composition (RFC M3)#1673
wprazuch wants to merge 2 commits into
mainfrom
wprazuch/config-composer

wprazuch commented Jun 23, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

wprazuch commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wprazuch commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What it does

Tests

Uh oh!

copy-pr-bot Bot commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

wprazuch commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wprazuch commented Jun 23, 2026 •

edited

Loading