feat(cli): add 'gym env validate' pre-flight config check (#1205 friction #12) (#1599)

## What

Adds **`gym env validate`** (+ `ng_validate` / `nemo_gym_validate`
deprecated shims) — runs the full config parse with **no Ray and no
server subprocesses**, then exits **0 (valid) / 1 (invalid)** with a
clean, rich-escaped message (**no traceback**). Returns in well under a
second instead of after a ~30–60s Ray bootstrap.

```bash
gym env validate --config resources_servers/<env>/configs/<env>.yaml --config responses_api_models/<model>/configs/<model>.yaml
gym env validate --benchmark gsm8k --model-type openai_model
```

## How

`validate()` lives in `cli/env.py` and is registered as `env validate`
in the `gym` router (`cli/main.py` COMMANDS) with the same
config-selection flags as `env start` (`--config`, `--benchmark`,
`--environment`, `--resources-server`, `--model-type`, `--search-dir`,
`--model*`). It reuses the same `get_global_config_dict()` parse path
the other commands use, so the validation checks stay in sync:

- **config_paths** resolution — missing/typo'd
([#1488](#1488)) and malformed
([#1490](#1490))
- **server cross-references** — unknown `name:` refs
([#1561](#1561))
- **mandatory `???`** values
([#1575](#1575))
- **schema** (`BaseNeMoGymCLIConfig`)

Wrapped in `exit_cleanly_on_config_error` (from #1609) so any
`ConfigError` becomes a clean message + `exit 1`. A dummy `policy_model`
is injected (the `NO_MODEL` parser config, as in `gym list` / `env
compose`) so model interpolations like `${policy_base_url}` resolve
without real creds — validation is about config **well-formedness**; the
real model is supplied by the `--model*` flags at run time.

## Targets `main`

Originally drafted on the unified-CLI epic branch; rebuilt directly on
`main` now that [#1630](#1630)
(and #1637/#1609/#1635/#1671) have merged. The old branch contents (a
snapshot of the CLI refactor + unrelated CI commits) were superseded and
replaced.

## Scope note

The zero-server check
([#1489](#1489), "nothing
configured to run") is intentionally **not** part of `validate`:
`NO_MODEL` injects a dummy model server (which would defeat the check),
and "is anything configured to run" is a *start*-time concern already
enforced by `gym env start` before Ray init. `validate` focuses on
config well-formedness.

## Why

Epic [#1205](#1205) friction
#12 (no config validation tooling) — the M1 "fast failure triage"
deliverable. Config errors otherwise only surface after Ray starts
(~30–60s).

## Tests

- `test_cli_main.py`: `gym env validate --config X` routes to
`nemo_gym.cli.env:validate` with `+config_paths=[X]` (added to the
parametrized config-command matrix).
- `test_cli.py`: `validate()` prints OK on a valid config; a raised
`ConfigError` becomes `exit 1` (no traceback).
- All `test_cli` + `test_cli_main` + `test_cli_legacy` pass (the only
failures are the pre-existing Python-3.12 `TestDidYouMean` argparse
issue on `main`); ruff + pre-commit clean. Smoke-tested end-to-end: `✓
Config is valid.` on a real benchmark, clean error + `exit 1` on a bad
path, and the `ng_validate` deprecation shim.

---------

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Author: wprazuch

nemo_gym/cli/env.py

@@ -864,6 +864,39 @@ def dump_config():  # pragma: no cover
     print(OmegaConf.to_yaml(global_config_dict, resolve=True))
 
 
+@exit_cleanly_on_config_error
+def validate():
+    """Validate a config without starting Ray or any server subprocess.
+
+    Runs the full config parse — config_paths resolution (missing/malformed), server cross-reference
+    validation, mandatory `???` values, and schema — then exits 0 (valid) or, via
+    `exit_cleanly_on_config_error`, 1 with a clean traceback-free message. No Ray, no servers, so it
+    returns in well under a second instead of after Ray bootstrap.
+
+    No model config is required: a dummy `policy_model` is injected (the `NO_MODEL` parser config, as
+    in `gym list` / `env compose`) so model interpolations (e.g. `${policy_base_url}`) resolve —
+    validation is about config well-formedness, not the model. Pass a model config / `--model-type`
+    as well if you want it validated too.
+
+    Examples:
+
+    ```bash
+    gym env validate --environment <env>
+    gym env validate --benchmark <benchmark>
+    # or by explicit config path(s):
+    gym env validate --config resources_servers/<env>/configs/<env>.yaml
+    ```
+    """
+    global_config_dict = get_global_config_dict(
+        global_config_dict_parser_config=GlobalConfigDictParserConfig(
+            initial_global_config_dict=GlobalConfigDictParserConfig.NO_MODEL_GLOBAL_CONFIG_DICT,
+        ),
+    )
+    BaseNeMoGymCLIConfig.model_validate(global_config_dict)
+
+    rich.print("[green]✓[/green] Config is valid.")
+
+
 def list_environments() -> None:
     """List the environments available under environments/, by short name.
 

nemo_gym/cli/legacy.py

@@ -48,6 +48,7 @@
     "gitlab_to_hf_dataset": ["dataset", "migrate"],
     "delete_dataset_from_gitlab": ["dataset", "rm"],
     "dump_config": ["env", "resolve"],
+    "validate": ["env", "validate"],
     "help": ["--help"],
     "status": ["env", "status"],
     "pip_list": ["env", "packages"],

nemo_gym/cli/main.py

@@ -397,6 +397,21 @@ def _dataset_download(args: argparse.Namespace, overrides: list[str]) -> None:
         summary="Resolve the final config from configs, flags, and overrides.",
         flags=(CONFIG,),
     ),
+    "env validate": Command(
+        target="nemo_gym.cli.env:validate",
+        summary="Validate a config (paths, cross-refs, ??? values, servers) fast — no Ray, no servers.",
+        flags=(
+            CONFIG,
+            BENCHMARK,
+            ENVIRONMENT,
+            RESOURCES_SERVER_CONFIG,
+            MODEL_TYPE,
+            SEARCH_DIR,
+            MODEL,
+            MODEL_URL,
+            MODEL_API_KEY,
+        ),
+    ),
     "env packages": Command(
         target="nemo_gym.cli.env:pip_list",
         summary="Print pip packages for the selected resources server.",

nemo_gym/config_types.py

@@ -137,10 +137,6 @@ def is_server_ref(config_dict: DictConfig) -> Optional[ServerRef]:
         return None
 
 
-class ServerRefNotFoundError(ValueError):
-    """A server cross-reference points to an instance that is not defined in the merged config."""
-
-
 class ConfigError(Exception):
     """Base for user-facing configuration errors.
 
@@ -167,6 +163,10 @@ class ConfigMissingValuesError(ConfigError, ValueError):
     """One or more required config values are still unset (OmegaConf '???') after merging."""
 
 
+class ServerRefNotFoundError(ConfigError, ValueError):
+    """A server cross-reference points to an instance that is not defined in the merged config."""
+
+
 ########################################
 # Dataset configs for handling and upload/download
 ########################################

pyproject.toml

@@ -406,6 +406,9 @@ ng_delete_dataset_from_gitlab = "nemo_gym.cli.legacy:main"
 nemo_gym_dump_config = "nemo_gym.cli.legacy:main"
 ng_dump_config = "nemo_gym.cli.legacy:main"
 
+nemo_gym_validate = "nemo_gym.cli.legacy:main"
+ng_validate = "nemo_gym.cli.legacy:main"
+
 # Display help
 nemo_gym_help = "nemo_gym.cli.legacy:main"
 ng_help = "nemo_gym.cli.legacy:main"

tests/unit_tests/test_cli.py

@@ -14,6 +14,7 @@
 # limitations under the License.
 import json
 import shutil
+import sys
 import tomllib
 from importlib import import_module
 from pathlib import Path
@@ -35,6 +36,7 @@
     exit_cleanly_on_config_error,
     init_resources_server,
     list_environments,
+    validate,
 )
 from nemo_gym.config_types import ConfigError, NoServerInstancesError, ResourcesServerInstanceConfig
 from nemo_gym.registry import EnvironmentEntry
@@ -290,6 +292,53 @@ def test_config_error_base_catches_subclasses(self) -> None:
         assert issubclass(NoServerInstancesError, ConfigError)
 
 
+class TestValidate:
+    def _validate_config(self, monkeypatch: MonkeyPatch, tmp_path: Path, config_yaml: str) -> None:
+        """Run the real `validate()` against a config file (no mocking of the parse path)."""
+        config_file = tmp_path / "config.yaml"
+        config_file.write_text(config_yaml)
+        # chdir to a clean dir so a repo-local env.yaml isn't picked up; clear the parse cache.
+        monkeypatch.chdir(tmp_path)
+        monkeypatch.setattr(sys, "argv", ["gym", f"+config_paths=[{config_file}]"])
+        monkeypatch.setattr(nemo_gym.global_config, "_GLOBAL_CONFIG_DICT", None)
+        validate()
+
+    def test_valid_config_passes(self, monkeypatch: MonkeyPatch, tmp_path, capsys) -> None:
+        # A well-formed config (real parse, real checks) -> "valid".
+        self._validate_config(
+            monkeypatch,
+            tmp_path,
+            "my_server:\n  resources_servers:\n    my_server:\n      entrypoint: app.py\n      domain: other\n",
+        )
+        assert "valid" in capsys.readouterr().out.lower()
+
+    def test_unknown_cross_reference_exits_nonzero(self, monkeypatch: MonkeyPatch, tmp_path) -> None:
+        # An agent referencing a resources server that isn't defined -> ServerRefNotFoundError ->
+        # clean exit(1) (real cross-reference validation, not a mock).
+        bad = (
+            "my_agent:\n"
+            "  responses_api_agents:\n"
+            "    a:\n"
+            "      entrypoint: app.py\n"
+            "      resources_server:\n        type: resources_servers\n        name: does_not_exist\n"
+            "      model_server:\n        type: responses_api_models\n        name: policy_model\n"
+        )
+        with raises(SystemExit) as exc_info:
+            self._validate_config(monkeypatch, tmp_path, bad)
+        assert exc_info.value.code == 1
+
+    def test_config_error_becomes_clean_exit(self, monkeypatch: MonkeyPatch) -> None:
+        # Fast unit check of the decorator path: any ConfigError -> exit(1), no traceback.
+        def _raise(**kwargs):
+            raise NoServerInstancesError("nothing configured to run")
+
+        monkeypatch.setattr(nemo_gym.cli.env, "get_global_config_dict", _raise)
+
+        with raises(SystemExit) as exc_info:
+            validate()
+        assert exc_info.value.code == 1
+
+
 class TestListEnvironments:
     _ALPHA = EnvironmentEntry(
         name="alpha",

tests/unit_tests/test_cli_main.py

@@ -54,6 +54,7 @@ def _split_overrides(overrides: list[str]) -> tuple[set[str], set[str]]:
 CONFIG_COMMANDS = [
     (["env", "start"], "nemo_gym.cli.env:run"),
     (["env", "resolve"], "nemo_gym.cli.env:dump_config"),
+    (["env", "validate"], "nemo_gym.cli.env:validate"),
     (["eval", "prepare"], "nemo_gym.cli.eval:prepare_benchmark"),
     (["eval", "aggregate"], "nemo_gym.cli.eval:aggregate_rollouts"),
     (["eval", "run"], "nemo_gym.cli.eval:e2e_rollout_collection"),