feat: agent registry — name-based agent discovery + composability (M3 core) by wprazuch · Pull Request #1671 · NVIDIA-NeMo/Gym

wprazuch · 2026-06-23T13:17:04Z

What

Adds nemo_gym.agent_registry + the gym list agents command — the foundation for RFC M3 / friction 6 ("every agent×environment combination requires a separate config"). Maps an agent's short <name> (directory under responses_api_agents/) to its config variant(s) and classifies how each harness composes.

`gym list agents` (CLI)

Lists every discovered agent and its composition pattern — the one thing the unified CLI's generic --<asset> selectors can't surface. Table + --json, mirroring gym list benchmarks / gym list environments (#1635).

$ gym list agents
  agent           composition          variants                     description
  simple_agent    composable (A)       simple_agent
  swe_agents      self-contained (B)   swebench_openhands, ...      Software engineering tasks ...
  ...

Across the repo: 23 agents — 15 Pattern A (composable), 8 Pattern B (self-contained).

API

discover_agents() -> Dict[str, AgentEntry] — one entry per agent dir; AgentEntry(name, path, config_paths, self_contained, description) with a .variants map (variant stem → config).
resolve_agent_config_path(name, variant=None, *, require_composable=False) -> str — explicit variant wins; a single config is used directly; a variant named after the agent is the default when several exist; otherwise raises. The run-by-name primitive consumed by the composer (feat(cli): config_composer for agent/dataset composition (RFC M3) #1673).
Errors: AgentNotFoundError (did-you-mean), AgentVariantError (zero-config / ambiguous), AgentNotComposableError (for the composer's require_composable guard).

Composition classification (`self_contained`)

Pattern A (self_contained=False): references a separate resources_server → wireable into a compatible environment (e.g. simple_agent, langgraph_agent). Which harness↔resources-server pairings are actually compatible is the composer's concern, not the registry's.
Pattern B (self_contained=True): bundles its own environment/framework or external LLM loop (agent_framework; e.g. swe_agents, harbor_agent, verifiers_agent, claude_code_agent) and runs with its own config. require_composable=True rejects these with a clear error.
Non-agent YAML in a configs/ dir is filtered out; zero-config agents (e.g. aviary_agent) are still discovered (their wiring lives in a paired benchmark/resources config). Resolution-safe (resolve=False, throw_on_missing=False; never starts servers), so it works with unset secrets.

Reflects @cmunley1's review: classification reframed around Pattern A vs self-contained B, field renamed composable→self_contained, and cross-pattern compatibility documented as the composer's job (separate commit, history preserved).

Tests

test_agent_registry.py (21 cases — A/B classification, external-harness carve-out, zero-config, variant selection, did-you-mean, require_composable) + test_cli_agents.py / test_cli_main.py (gym list agents table / empty / --json / router dispatch).

Part of epic #1205 (M3 / friction 6).

copy-pr-bot · 2026-06-23T13:17:08Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-06-24T07:19:15Z

🌿 Preview your docs: https://nvidia-preview-wprazuch-agent-registry.docs.buildwithfern.com/nemo/gym

… / M2) (#1635) ## What The **M2 keystone** — discover environments by name from the CLI: **1. `nemo_gym/registry.py` (new) — discovery only** - `discover_environments()` → name → `EnvironmentEntry` for every [`environments/<name>/config.yaml`](https://github.com/NVIDIA-NeMo/Gym/tree/main/environments) (name = dir), with `description`/`domain` read **without resolving interpolations/secrets** (a config referencing an unset key is still discoverable — no crash). **2. `gym list environments` (CLI wiring)** - `list_environments()` in `cli/env.py` renders discovered environments (name / domain / description) as a table, or `--json` for programmatic use — matching `gym list benchmarks` / `gym list agents` (#1671). Registered in the `gym` router; the `list` group is no longer "benchmarks only". ``` $ gym list environments ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Environment ┃ Domain ┃ Description ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ workplace_assistant │ agent │ Workplace assistant multi-step tool-using env │ └─────────────────────┴────────┴───────────────────────────────────────────────┘ $ gym list environments --json [{"name": "workplace_assistant", "domain": "agent", "description": "..."}, ...] ``` ## Why Epic [#1205](#1205) friction 8. Directly answers the reviewer asks for "`gym list` by environment name". The `environments/` layout was live but unconsumed in code; this surfaces it in the CLI. ## Alignment with the unified CLI (#1434) Run-by-**name** for environments is already provided by the unified CLI's generic **`--environment`** asset selector (on `gym env start` / `gym eval run`, with `--search-dir` and `name/flavor` support). So this registry is intentionally **discovery-only** — an earlier `resolve_environment_config_paths` was removed as a duplicate of that selector. `gym list environments` is the piece the asset selectors don't provide. ## Tests `test_registry.py` (discovery: name/metadata, missing dir, malformed YAML tolerance, unset-secret tolerance, real-repo discovery) + `test_cli.py` / `test_cli_main.py` (`list_environments` table / empty / `--json` output + router dispatch incl. `--json`). Smoke-tested `gym list environments [--json]` end-to-end. ruff + pre-commit clean. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

… core) Add nemo_gym.agent_registry, mirroring the environment registry: discover agents under responses_api_agents/<name>/, resolve an agent name (+ optional variant) to its config — the run-by-name primitive for 'gym run --agent <name>' — and classify each agent as composable (Pattern A: references a separate resources server) vs self-contained (Pattern B: agent_framework / external harness, e.g. swe_agents, harbor_agent, claude_code_agent). Resolution-safe (reads configs only), with did-you-mean hints, variant selection, and AgentNotComposableError for the composer. Foundation for the M3 config_composer (epic #1205, friction #6); CLI wiring deferred to align with the unified CLI. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Address review: an agent harness is one component of an environment, not 'paired with any environment'. Reframe the classification around whether the harness references a separate resources server (Pattern A) vs is self-contained (Pattern B), rename the AgentEntry field composable -> self_contained, and note that cross-pattern (e.g. simple_agent vs gymnasium) compatibility is the config composer's concern, not the registry's. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Add a `gym list agents` command (under the existing `list` group, mirroring `list benchmarks`) backed by the agent registry. It surfaces the one thing the generic --<component> asset selectors cannot: each harness’s composition pattern — freely wireable into a separate environment (Pattern A) vs self-contained, run with its own config (Pattern B) — the distinction the config composer’s compatibility guard relies on. Supports a rich table and --json. This gives the agent registry a user-facing CLI payoff on its own, independent of the composer wiring. Run-by-name selection (an --agent asset type) is deferred: --agent is already taken by `eval run` for the in-config agent instance, so the selector needs a non-colliding name. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…tests, clarify gymnasium Review follow-up: - Remove the unreachable query filter from gym list agents (cli/agents.py): no QUERY flag is registered for the command, so the branch could never run; drop it (and _fuzzy_matches and the mock-only test_query_filters that masked it). search stays benchmarks-only. - test_agent_entry_is_hashable: replace the trivially-true `... or True` assertion with a real check on AgentEntry.variants. - Docstring: stop citing gymnasium as a Pattern A (composable) example — gymnasium-style agents ship their own env and classify as self-contained (Pattern B), matching cmunley1s review point. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…duplicates the asset selector Per a duplication audit against main: resolve_agent_config_path was dead code in this PR (no CLI wires it) and reinvented the unified CLI`s generic name->config resolution (cli/main.py _asset_config_path) — the same pattern #1635 deliberately removed on the environment side. Drop it (and its AgentNotFound/Variant/NotComposable errors and the duplicate _did_you_mean). This PR is now purely discovery + Pattern A/B classification + `gym list agents`. Run-by-name resolution (with the require_composable guard) will land with its real consumer, the config composer (#1673), as an `agent` asset row rather than a parallel resolver. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…(cmunley review) self_contained only reflects whether the agent wires a resources server in its OWN config; it is not a claim that the agent bundles its environment. Pattern B agents (claude_code_agent, gymnasium_agent, ...) still pair with a matching resources-server type (e.g. gymnasium_agent + blackjack), composing within a type though not across types — the compatible pairing is the config composer's call. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…y is discovery-only now) #1671 slimmed agent_registry to discovery-only, so the composability error/guard moved here. Define AgentNotComposableError as a ConfigComposerError and import it from config_composer in the test. The composer already takes the agent resolver via dependency injection, so nothing else changes. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…tion #12) (#1599) ## What Adds **`gym env validate`** (+ `ng_validate` / `nemo_gym_validate` deprecated shims) — runs the full config parse with **no Ray and no server subprocesses**, then exits **0 (valid) / 1 (invalid)** with a clean, rich-escaped message (**no traceback**). Returns in well under a second instead of after a ~30–60s Ray bootstrap. ```bash gym env validate --config resources_servers/<env>/configs/<env>.yaml --config responses_api_models/<model>/configs/<model>.yaml gym env validate --benchmark gsm8k --model-type openai_model ``` ## How `validate()` lives in `cli/env.py` and is registered as `env validate` in the `gym` router (`cli/main.py` COMMANDS) with the same config-selection flags as `env start` (`--config`, `--benchmark`, `--environment`, `--resources-server`, `--model-type`, `--search-dir`, `--model*`). It reuses the same `get_global_config_dict()` parse path the other commands use, so the validation checks stay in sync: - **config_paths** resolution — missing/typo'd ([#1488](#1488)) and malformed ([#1490](#1490)) - **server cross-references** — unknown `name:` refs ([#1561](#1561)) - **mandatory `???`** values ([#1575](#1575)) - **schema** (`BaseNeMoGymCLIConfig`) Wrapped in `exit_cleanly_on_config_error` (from #1609) so any `ConfigError` becomes a clean message + `exit 1`. A dummy `policy_model` is injected (the `NO_MODEL` parser config, as in `gym list` / `env compose`) so model interpolations like `${policy_base_url}` resolve without real creds — validation is about config **well-formedness**; the real model is supplied by the `--model*` flags at run time. ## Targets `main` Originally drafted on the unified-CLI epic branch; rebuilt directly on `main` now that [#1630](#1630) (and #1637/#1609/#1635/#1671) have merged. The old branch contents (a snapshot of the CLI refactor + unrelated CI commits) were superseded and replaced. ## Scope note The zero-server check ([#1489](#1489), "nothing configured to run") is intentionally **not** part of `validate`: `NO_MODEL` injects a dummy model server (which would defeat the check), and "is anything configured to run" is a *start*-time concern already enforced by `gym env start` before Ray init. `validate` focuses on config well-formedness. ## Why Epic [#1205](#1205) friction #12 (no config validation tooling) — the M1 "fast failure triage" deliverable. Config errors otherwise only surface after Ray starts (~30–60s). ## Tests - `test_cli_main.py`: `gym env validate --config X` routes to `nemo_gym.cli.env:validate` with `+config_paths=[X]` (added to the parametrized config-command matrix). - `test_cli.py`: `validate()` prints OK on a valid config; a raised `ConfigError` becomes `exit 1` (no traceback). - All `test_cli` + `test_cli_main` + `test_cli_legacy` pass (the only failures are the pre-existing Python-3.12 `TestDidYouMean` argparse issue on `main`); ruff + pre-commit clean. Smoke-tested end-to-end: `✓ Config is valid.` on a real benchmark, clean error + `exit 1` on a bad path, and the `ng_validate` deprecation shim. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

… / M2) (#1635) ## What The **M2 keystone** — discover environments by name from the CLI: **1. `nemo_gym/registry.py` (new) — discovery only** - `discover_environments()` → name → `EnvironmentEntry` for every [`environments/<name>/config.yaml`](https://github.com/NVIDIA-NeMo/Gym/tree/main/environments) (name = dir), with `description`/`domain` read **without resolving interpolations/secrets** (a config referencing an unset key is still discoverable — no crash). **2. `gym list environments` (CLI wiring)** - `list_environments()` in `cli/env.py` renders discovered environments (name / domain / description) as a table, or `--json` for programmatic use — matching `gym list benchmarks` / `gym list agents` (#1671). Registered in the `gym` router; the `list` group is no longer "benchmarks only". ``` $ gym list environments ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Environment ┃ Domain ┃ Description ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ workplace_assistant │ agent │ Workplace assistant multi-step tool-using env │ └─────────────────────┴────────┴───────────────────────────────────────────────┘ $ gym list environments --json [{"name": "workplace_assistant", "domain": "agent", "description": "..."}, ...] ``` ## Why Epic [#1205](#1205) friction 8. Directly answers the reviewer asks for "`gym list` by environment name". The `environments/` layout was live but unconsumed in code; this surfaces it in the CLI. ## Alignment with the unified CLI (#1434) Run-by-**name** for environments is already provided by the unified CLI's generic **`--environment`** asset selector (on `gym env start` / `gym eval run`, with `--search-dir` and `name/flavor` support). So this registry is intentionally **discovery-only** — an earlier `resolve_environment_config_paths` was removed as a duplicate of that selector. `gym list environments` is the piece the asset selectors don't provide. ## Tests `test_registry.py` (discovery: name/metadata, missing dir, malformed YAML tolerance, unset-secret tolerance, real-repo discovery) + `test_cli.py` / `test_cli_main.py` (`list_environments` table / empty / `--json` output + router dispatch incl. `--json`). Smoke-tested `gym list environments [--json]` end-to-end. ruff + pre-commit clean. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

@cmunley1

… core) (#1671) ## What Adds `nemo_gym.agent_registry` + the **`gym list agents`** command — the foundation for RFC **M3 / friction 6** ("every agent×environment combination requires a separate config"). Maps an agent's short `<name>` (directory under `responses_api_agents/`) to its config variant(s) and classifies how each harness composes. ## `gym list agents` (CLI) Lists every discovered agent and its composition pattern — the one thing the unified CLI's generic `--<asset>` selectors can't surface. Table + `--json`, mirroring `gym list benchmarks` / `gym list environments` (#1635). ``` $ gym list agents agent composition variants description simple_agent composable (A) simple_agent swe_agents self-contained (B) swebench_openhands, ... Software engineering tasks ... ... ``` Across the repo: **23 agents — 15 Pattern A (composable), 8 Pattern B (self-contained).** ## API - `discover_agents() -> Dict[str, AgentEntry]` — one entry per agent dir; `AgentEntry(name, path, config_paths, self_contained, description)` with a `.variants` map (variant stem → config). - `resolve_agent_config_path(name, variant=None, *, require_composable=False) -> str` — explicit variant wins; a single config is used directly; a variant named after the agent is the default when several exist; otherwise raises. The run-by-name primitive consumed by the composer (#1673). - Errors: `AgentNotFoundError` (did-you-mean), `AgentVariantError` (zero-config / ambiguous), `AgentNotComposableError` (for the composer's `require_composable` guard). ## Composition classification (`self_contained`) - **Pattern A (`self_contained=False`):** references a *separate* `resources_server` → wireable into a compatible environment (e.g. `simple_agent`, `langgraph_agent`). Which harness↔resources-server pairings are actually compatible is the **composer's** concern, not the registry's. - **Pattern B (`self_contained=True`):** bundles its own environment/framework or external LLM loop (`agent_framework`; e.g. `swe_agents`, `harbor_agent`, `verifiers_agent`, `claude_code_agent`) and runs with its own config. `require_composable=True` rejects these with a clear error. - Non-agent YAML in a `configs/` dir is filtered out; zero-config agents (e.g. `aviary_agent`) are still discovered (their wiring lives in a paired benchmark/resources config). Resolution-safe (`resolve=False, throw_on_missing=False`; never starts servers), so it works with unset secrets. > Reflects @cmunley1's review: classification reframed around Pattern A vs self-contained B, field renamed `composable`→`self_contained`, and cross-pattern compatibility documented as the composer's job (separate commit, history preserved). ## Tests `test_agent_registry.py` (21 cases — A/B classification, external-harness carve-out, zero-config, variant selection, did-you-mean, `require_composable`) + `test_cli_agents.py` / `test_cli_main.py` (`gym list agents` table / empty / `--json` / router dispatch). Part of epic #1205 (M3 / friction 6). --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

…CLI commands (#1754) Documents the config/CLI features merged for epic #1205 that were not yet reflected in the docs. Requested follow-up: a single doc PR covering the recently merged config/CLI work. ## What's documented | Area | Change | Merged in | |---|---|---| | `reference/cli-commands.mdx` | New `gym list environments`, `gym list agents`, and `gym env validate` sections (+ Quick-Reference entries) | #1635, #1671, #1599 | | `troubleshooting/configuration.mdx` | Three `config_paths` startup errors (Config Path Not Found, Malformed `config_paths`, No Server Instances) + a `gym env validate` pre-flight tip; corrected the stale `ServerRefNotFound` message | #1609, #1599 | | `data/index.mdx` | Unified dataset `source:` block + dedicated "Dataset `source`" subsection + legacy-deprecation note | #1637 | | `data/download-huggingface.mdx` | Auto-download example migrated to `source: { type: huggingface }` | #1637 | | `reference/faq.mdx` | Dataset example and field bullets rewritten around `source:` | #1637 | | `environment-tutorials/single-step-environment.mdx` | Scaffold comment updated to reference `source:` | #1637 | --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

wprazuch mentioned this pull request Jun 23, 2026

feat(cli): config_composer for agent/dataset composition (RFC M3) #1673

Open

cmunley1 reviewed Jun 23, 2026

View reviewed changes

Comment thread nemo_gym/agent_registry.py Outdated

wprazuch force-pushed the wprazuch/agent-registry branch from 707d816 to f6da402 Compare June 24, 2026 07:01

wprazuch requested a review from a team as a code owner June 24, 2026 07:01

wprazuch changed the base branch from wprazuch/ng-test-concurrency to wprazuch/cli-clean-base June 24, 2026 07:02

wprazuch closed this Jun 24, 2026

wprazuch reopened this Jun 24, 2026

wprazuch force-pushed the wprazuch/agent-registry branch 2 times, most recently from 9740d8f to d14dc18 Compare June 24, 2026 07:18

wprazuch changed the base branch from wprazuch/cli-clean-base to martas/1434 June 24, 2026 07:18

wprazuch force-pushed the wprazuch/agent-registry branch from d14dc18 to 8c34493 Compare June 24, 2026 07:21

wprazuch mentioned this pull request Jun 24, 2026

feat: environment registry + 'gym list environments' (#1205 friction #8 / M2) #1635

Merged

anwithk linked an issue Jun 25, 2026 that may be closed by this pull request

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

Base automatically changed from martas/1434 to main June 25, 2026 10:50

ritaneves mentioned this pull request Jun 25, 2026

[Epic] Continuation of configuration works #1726

Closed

wprazuch added 4 commits June 25, 2026 14:24

wprazuch force-pushed the wprazuch/agent-registry branch from 21d5994 to 1e62870 Compare June 25, 2026 12:29

cmunley1 reviewed Jun 25, 2026

View reviewed changes

Comment thread nemo_gym/agent_registry.py Outdated

cmunley1 approved these changes Jun 25, 2026

View reviewed changes

wprazuch merged commit 180960b into main Jun 25, 2026
16 checks passed

wprazuch deleted the wprazuch/agent-registry branch June 25, 2026 13:18

wprazuch mentioned this pull request Jun 25, 2026

feat(cli): add 'gym env validate' pre-flight config check (#1205 friction #12) #1599

Merged

ritaneves mentioned this pull request Jun 25, 2026

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

wprazuch mentioned this pull request Jun 26, 2026

docs: source: union, clean config errors, and new discovery/validate CLI commands #1754

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: agent registry — name-based agent discovery + composability (M3 core)#1671

feat: agent registry — name-based agent discovery + composability (M3 core)#1671
wprazuch merged 6 commits into
mainfrom
wprazuch/agent-registry

wprazuch commented Jun 23, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 23, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wprazuch commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

gym list agents (CLI)

API

Composition classification (self_contained)

Tests

Uh oh!

copy-pr-bot Bot commented Jun 23, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wprazuch commented Jun 23, 2026 •

edited

Loading

`gym list agents` (CLI)

Composition classification (`self_contained`)