Gym CLI refactor

.claude/skills/add-benchmark/SKILL.md

@@ -32,10 +32,10 @@ Before starting, determine which type of benchmark you're adding:
 
 ### Step 1: Scaffold the server
 
-Run `ng_init_resources_server` to generate the directory structure:
+Run `gym env init` to generate the directory structure:
 
 ```bash
-ng_init_resources_server +entrypoint=resources_servers/my_benchmark
+gym env init --resources-server my_benchmark
 ```
 
 This creates:
@@ -77,10 +77,10 @@ Convert your source dataset to Gym JSONL format. Each line must have `responses_
 **`train`/`validation` datasets**: Upload to the GitLab dataset registry — these must NOT be committed to git.
 
 ```bash
-ng_upload_dataset_to_gitlab \
-    +dataset_name=my_benchmark \
-    +version=0.0.1 \
-    +input_jsonl_fpath=resources_servers/my_benchmark/data/my_dataset.jsonl
+gym dataset upload --storage gitlab \
+    --name my_benchmark \
+    --revision 0.0.1 \
+    --input resources_servers/my_benchmark/data/my_dataset.jsonl
 ```
 
 Requires MLflow credentials in `env.yaml` (or passed via CLI):
@@ -94,12 +94,16 @@ mlflow_tracking_token: <your-gitlab-api-token>
 **Validate** your data:
 ```bash
 # Validate example data (for PR submission)
-ng_prepare_data "+config_paths=[resources_servers/my_benchmark/configs/my_benchmark.yaml]" \
-    +output_dirpath=/tmp/prepare +mode=example_validation
+gym dataset collate --config resources_servers/my_benchmark/configs/my_benchmark.yaml \
+    --output-dir /tmp/prepare \
+    --mode example_validation
 
 # Download and prepare train/validation from GitLab
-ng_prepare_data "+config_paths=[resources_servers/my_benchmark/configs/my_benchmark.yaml]" \
-    +output_dirpath=data/my_benchmark +mode=train_preparation +should_download=true +data_source=gitlab
+gym dataset collate --config resources_servers/my_benchmark/configs/my_benchmark.yaml \
+    --output-dir data/my_benchmark \
+    --mode train_preparation \
+    --download \
+    +data_source=gitlab
 ```
 
 ### Step 3: Implement verify()
@@ -162,7 +166,7 @@ Both fields must coexist: `jsonl_fpath` is the local download destination, `gitl
 
 ```bash
 # Run server tests (creates isolated .venv, slow on first run)
-ng_test +entrypoint=resources_servers/my_benchmark
+gym env test --resources-server my_benchmark
 
 # Run core library tests to check nothing broke
 pytest tests/unit_tests/ -x
@@ -174,14 +178,18 @@ Test coverage must be >= 95%. Write tests for: verify pass, verify fail (wrong o
 
 ```bash
 # Start servers
-ng_run "+config_paths=[resources_servers/my_benchmark/configs/my_benchmark.yaml,responses_api_models/openai_model/configs/openai_model.yaml]"
+gym env start \
+    --config resources_servers/my_benchmark/configs/my_benchmark.yaml \
+    --model-type openai_model
 
 # Quick test with example data
-ng_collect_rollouts +agent_name=my_benchmark_simple_agent \
-  +input_jsonl_fpath=resources_servers/my_benchmark/data/example.jsonl \
-  +output_jsonl_fpath=results/example_rollouts.jsonl \
-  +num_repeats=1 \
-  "+responses_create_params={max_output_tokens: 16384, temperature: 1.0}"
+gym eval run --no-serve \
+  --agent my_benchmark_simple_agent \
+  --input resources_servers/my_benchmark/data/example.jsonl \
+  --output results/example_rollouts.jsonl \
+  --num-repeats 1 \
+  --max-output-tokens 16384 \
+  --temperature 1.0
 
 # Inspect results
 ```
@@ -196,17 +204,17 @@ Run against multiple models to validate correctness. Recommended suite:
 
 ```bash
 # Collect rollouts
-ng_collect_rollouts +agent_name=my_benchmark_simple_agent \
-  +input_jsonl_fpath=resources_servers/my_benchmark/data/my_dataset.jsonl \
-  +output_jsonl_fpath=results/rollouts.jsonl \
-  +num_repeats=5 \
-  "+responses_create_params={max_output_tokens: 16384, temperature: 1.0}"
+gym eval run --no-serve \
+  --agent my_benchmark_simple_agent \
+  --input resources_servers/my_benchmark/data/my_dataset.jsonl \
+  --output results/rollouts.jsonl \
+  --num-repeats 5 \
+  --max-output-tokens 16384 \
+  --temperature 1.0
 
 # Compute per-task pass rates
-ng_reward_profile +input_jsonl_fpath=resources_servers/my_benchmark/data/my_dataset.jsonl \
-  +rollouts_jsonl_fpath=results/rollouts.jsonl \
-  +output_jsonl_fpath=results/profiled.jsonl \
-  +pass_threshold=1.0
+gym eval profile --inputs resources_servers/my_benchmark/data/my_dataset.jsonl \
+  --rollouts results/rollouts.jsonl
 
 # Aggregate metrics (pass@1 = avg_reward, pass@k from max_reward)
 python scripts/print_aggregate_results.py +jsonl_fpath=results/profiled.jsonl

.claude/skills/add-benchmark/references/patterns.md

@@ -546,7 +546,7 @@ Externalize system prompts to text files, pass via `--system-prompt` argument. M
 
 ### `data/.gitignore` default patterns
 
-Generated by `ng_init_resources_server`:
+Generated by `gym env init`:
 ```
 *train.jsonl
 *validation.jsonl
@@ -562,10 +562,10 @@ If your filename doesn't match (e.g. `my_eval.jsonl`), add a custom pattern (e.g
 1. **Generate** the JSONL file using your conversion script (in the source repo)
 2. **Upload** to GitLab dataset registry:
    ```bash
-   ng_upload_dataset_to_gitlab \
-       +dataset_name=my_benchmark \
-       +version=0.0.1 \
-       +input_jsonl_fpath=resources_servers/my_benchmark/data/my_dataset.jsonl
+   gym dataset upload --storage gitlab \
+       --name my_benchmark \
+       --revision 0.0.1 \
+       --input resources_servers/my_benchmark/data/my_dataset.jsonl
    ```
 3. **Add `gitlab_identifier`** to the dataset entry in YAML config:
    ```yaml
@@ -591,18 +591,22 @@ mlflow_tracking_token: <your-gitlab-api-token>
 
 The tracking URI format is `https://<gitlab-host>/api/v4/projects/<PROJECT_ID>/ml/mlflow`.
 
-### Verification with `ng_prepare_data`
+### Verification with `gym dataset collate`
 
 Validate example data (for PR submission):
 ```bash
-ng_prepare_data "+config_paths=[resources_servers/my_benchmark/configs/my_benchmark.yaml]" \
-    +output_dirpath=/tmp/prepare +mode=example_validation
+gym dataset collate --config resources_servers/my_benchmark/configs/my_benchmark.yaml \
+    --output-dir /tmp/prepare \
+    --mode example_validation
 ```
 
 Download and prepare train/validation from GitLab:
 ```bash
-ng_prepare_data "+config_paths=[resources_servers/my_benchmark/configs/my_benchmark.yaml]" \
-    +output_dirpath=data/my_benchmark +mode=train_preparation +should_download=true +data_source=gitlab
+gym dataset collate --config resources_servers/my_benchmark/configs/my_benchmark.yaml \
+    --output-dir data/my_benchmark \
+    --mode train_preparation \
+    --download \
+    +data_source=gitlab
 ```
 
 ---

.claude/skills/nemo-gym-debugging/references/error-profiles.md

@@ -113,7 +113,7 @@ Next actions:
 
 Symptoms:
 
-- `ng_run` or `ng_collect_rollouts` fails before launching servers
+- `gym env start` or `gym eval run --no-serve` fails before launching servers
 - errors mention missing config keys, unknown overrides, or interpolation failures
 - wrong agent/resource server starts despite expected data
 
@@ -134,7 +134,7 @@ Evidence to collect:
 
 Next actions:
 
-- reduce to a minimal `ng_run "+config_paths=[...]"` command
+- reduce to a minimal `gym env start --config ...` command
 - add overrides back one at a time
 - trust target checkout code over copied command templates
 

.claude/skills/nemo-gym-debugging/references/request-boundary-visibility.md

@@ -9,7 +9,7 @@ This is an escalation ladder. Do not patch agents, resource servers, model adapt
 1. Enable Gym's existing request debug flag:
 
    ```bash
-   ng_run '+config_paths=[...]' ++global_aiohttp_client_request_debug=True
+   gym env start --config ... ++global_aiohttp_client_request_debug=True
    ```
 
 2. Re-run the smallest failing case and capture stdout/stderr.

.claude/skills/nemo-gym-reward-profiling/SKILL.md

@@ -2,7 +2,7 @@
 name: nemo-gym-reward-profiling
 description: >-
   Use to help users get started with Nemo Gym reward profiling. Covers the basic
-  ng_run, ng_collect_rollouts, and ng_reward_profile workflow, repeated rollouts,
+  gym env start, gym eval run, and gym eval profile workflow, repeated rollouts,
   materialized inputs, rollout JSONL artifacts, task and rollout identity, output
   inspection, partial profiling, and rollout_infos. For failed jobs, prefer
   nemo-gym-debugging.
@@ -14,16 +14,16 @@ description: >-
 
 Use this skill when the user wants to run, understand, or lightly modify Nemo Gym reward profiling. Keep the answer oriented around the normal workflow:
 
-`ng_run` starts model/resource servers, `ng_collect_rollouts` writes rollout artifacts, and `ng_reward_profile` generates profiling output from those artifacts.
+`gym env start` starts model/resources servers, `gym eval run --no-serve` writes rollout artifacts, and `gym eval profile` generates profiling output from those artifacts.
 
 If the user is primarily debugging a failed job or stack trace, use the `nemo-gym-debugging` skill first.
 
 ## Basic Workflow
 
 1. Identify the environment config paths and input JSONL.
-2. Start Gym servers with `ng_run`.
-3. Collect rollouts with `ng_collect_rollouts`; this writes `rollouts.jsonl` and `*_materialized_inputs.jsonl`.
-4. Run `ng_reward_profile` on the materialized inputs and rollout JSONL to generate `*_reward_profiling.jsonl`.
+2. Start Gym servers with `gym env start`.
+3. Collect rollouts with `gym eval run --no-serve`; this writes `rollouts.jsonl` and `*_materialized_inputs.jsonl`.
+4. Run `gym eval profile` on the materialized inputs and rollout JSONL to generate `*_reward_profiling.jsonl`.
 5. Inspect line counts and profile rows.
 
 Repeated rollouts are the main profiling lever. `num_repeats=1` is valid, but per-task averages and variance are only meaningful with multiple rollouts per task.
@@ -48,6 +48,6 @@ Load references only when the user needs that detail:
 
 ## Practical Defaults
 
-- Treat `ng_reward_profile` as the reward profiling step; rollout collection does not write reward profile files.
+- Treat `gym eval profile` as the reward profiling step; rollout collection does not write reward profile files.
 - Run strict profiling by default. If rollout collection stopped early, use `++allow_partial_rollouts=True` to profile completed rollouts and drop original input rows with no completed rollout.
 - Trust the target checkout's CLI help and `nemo_gym/reward_profile.py` over memory if flags differ.

.claude/skills/nemo-gym-reward-profiling/references/output-format.md

@@ -6,7 +6,7 @@ Reward profiling is built around joining materialized inputs with rollout result
 
 - `*_materialized_inputs.jsonl`: expanded inputs after repeat expansion. Each row should have `_ng_task_index` and `_ng_rollout_index`.
 - `rollouts.jsonl`: completed rollout results. Each row should have matching `_ng_task_index` and `_ng_rollout_index`.
-- `*_reward_profiling.jsonl`: task-level summaries produced by `ng_reward_profile`.
+- `*_reward_profiling.jsonl`: task-level summaries produced by `gym eval profile`.
 - `*_agent_metrics.json`: agent/global aggregate metrics.
 
 ## Task Profile Rows
@@ -36,7 +36,7 @@ Full model responses stay in `rollouts.jsonl`. Join back to full rows with `(_ng
 
 ## Partial Profiles
 
-Strict profiling is the default. If materialized inputs and rollout results do not have the same rollout keys, `ng_reward_profile` fails and suggests:
+Strict profiling is the default. If materialized inputs and rollout results do not have the same rollout keys, `gym eval profile` fails and suggests:
 
 ```bash
 ++allow_partial_rollouts=True

.claude/skills/nemo-gym-reward-profiling/references/quick-start.md

@@ -5,8 +5,6 @@ Substitute environment-specific config paths, input data, model endpoint, and ou
 ## Minimal Flow
 
 ```bash
-CONFIG_PATHS="your_model_config_paths,your_env_config_paths"
-
 POLICY_MODEL_NAME="your_policy_model_name"
 POLICY_BASE_URL="your_policy_base_url"
 POLICY_ENDPOINT_KEY="your_policy_endpoint_key"
@@ -19,41 +17,43 @@ AGENT_NAME="your_agent_name"
 NUM_REPEATS=2
 NUM_SAMPLES_IN_PARALLEL=8
 
-ng_run "+config_paths=[$CONFIG_PATHS]" \
-    +policy_model_name="$POLICY_MODEL_NAME" \
-    +policy_base_url="$POLICY_BASE_URL" \
-    +policy_api_key="$POLICY_ENDPOINT_KEY" &
+gym env start \
+    --config your_model_config_paths \
+    --config your_env_config_paths \
+    --model "$POLICY_MODEL_NAME" \
+    --model-url "$POLICY_BASE_URL" \
+    --model-api-key "$POLICY_ENDPOINT_KEY" &
 NG_RUN_PID=$!
 trap 'kill "$NG_RUN_PID" 2>/dev/null || true' EXIT
 ./scripts/wait_for_servers.sh "$NG_RUN_PID"
 
 agent_args=()
 if [[ -n "$AGENT_NAME" ]]; then
-    agent_args=(+agent_name="$AGENT_NAME")
+    agent_args=(--agent "$AGENT_NAME")
 fi
 
-ng_collect_rollouts \
+gym eval run --no-serve \
     "${agent_args[@]}" \
-    +input_jsonl_fpath="$DATA_JSONL" \
-    +output_jsonl_fpath="$ROLLOUTS_JSONL" \
-    +num_repeats="$NUM_REPEATS" \
-    +num_samples_in_parallel="$NUM_SAMPLES_IN_PARALLEL"
-
-ng_reward_profile \
-    ++materialized_inputs_jsonl_fpath="$MATERIALIZED_JSONL" \
-    ++rollouts_jsonl_fpath="$ROLLOUTS_JSONL"
+    --input "$DATA_JSONL" \
+    --output "$ROLLOUTS_JSONL" \
+    --num-repeats "$NUM_REPEATS" \
+    --concurrency "$NUM_SAMPLES_IN_PARALLEL"
+
+gym eval profile \
+    --inputs "$MATERIALIZED_JSONL" \
+    --rollouts "$ROLLOUTS_JSONL"
 ```
 
 If rows already contain `agent_ref`, leave `AGENT_NAME` empty. Passing `+agent_name` supplies a default for rows without one.
 
 ## Partial Rollouts
 
-By default, `ng_reward_profile` expects every materialized input row to have a matching rollout row. If collection stopped early, profile the completed rollouts with:
+By default, `gym eval profile` expects every materialized input row to have a matching rollout row. If collection stopped early, profile the completed rollouts with:
 
 ```bash
-ng_reward_profile \
-    ++materialized_inputs_jsonl_fpath="$MATERIALIZED_JSONL" \
-    ++rollouts_jsonl_fpath="$ROLLOUTS_JSONL" \
+gym eval profile \
+    --inputs "$MATERIALIZED_JSONL" \
+    --rollouts "$ROLLOUTS_JSONL" \
     ++allow_partial_rollouts=True
 ```
 

.codex/skills/nemo-gym-debugging/references/error-profiles.md

@@ -113,7 +113,7 @@ Next actions:
 
 Symptoms:
 
-- `ng_run` or `ng_collect_rollouts` fails before launching servers
+- `gym env start` or `gym eval run --no-serve` fails before launching servers
 - errors mention missing config keys, unknown overrides, or interpolation failures
 - wrong agent/resource server starts despite expected data
 
@@ -134,7 +134,7 @@ Evidence to collect:
 
 Next actions:
 
-- reduce to a minimal `ng_run "+config_paths=[...]"` command
+- reduce to a minimal `gym env start --config ...` command
 - add overrides back one at a time
 - trust target checkout code over copied command templates
 

.codex/skills/nemo-gym-debugging/references/request-boundary-visibility.md

@@ -9,7 +9,7 @@ This is an escalation ladder. Do not patch agents, resource servers, model adapt
 1. Enable Gym's existing request debug flag:
 
    ```bash
-   ng_run '+config_paths=[...]' ++global_aiohttp_client_request_debug=True
+   gym env start --config ... ++global_aiohttp_client_request_debug=True
    ```
 
 2. Re-run the smallest failing case and capture stdout/stderr.

.codex/skills/nemo-gym-reward-profiling/SKILL.md

@@ -2,7 +2,7 @@
 name: nemo-gym-reward-profiling
 description: >-
   Use to help users get started with Nemo Gym reward profiling. Covers the basic
-  ng_run, ng_collect_rollouts, and ng_reward_profile workflow, repeated rollouts,
+  gym env start, gym eval run, and gym eval profile workflow, repeated rollouts,
   materialized inputs, rollout JSONL artifacts, task and rollout identity, output
   inspection, partial profiling, and rollout_infos. For failed jobs, prefer
   nemo-gym-debugging.
@@ -14,16 +14,16 @@ description: >-
 
 Use this skill when the user wants to run, understand, or lightly modify Nemo Gym reward profiling. Keep the answer oriented around the normal workflow:
 
-`ng_run` starts model/resource servers, `ng_collect_rollouts` writes rollout artifacts, and `ng_reward_profile` generates profiling output from those artifacts.
+`gym env start` starts model/resources servers, `gym eval run --no-serve` writes rollout artifacts, and `gym eval profile` generates profiling output from those artifacts.
 
-If the user is primarily debugging a failed job or stack trace, use `$nemo-gym-debugging` first.
+If the user is primarily debugging a failed job or stack trace, use the `nemo-gym-debugging` skill first.
 
 ## Basic Workflow
 
 1. Identify the environment config paths and input JSONL.
-2. Start Gym servers with `ng_run`.
-3. Collect rollouts with `ng_collect_rollouts`; this writes `rollouts.jsonl` and `*_materialized_inputs.jsonl`.
-4. Run `ng_reward_profile` on the materialized inputs and rollout JSONL to generate `*_reward_profiling.jsonl`.
+2. Start Gym servers with `gym env start`.
+3. Collect rollouts with `gym eval run --no-serve`; this writes `rollouts.jsonl` and `*_materialized_inputs.jsonl`.
+4. Run `gym eval profile` on the materialized inputs and rollout JSONL to generate `*_reward_profiling.jsonl`.
 5. Inspect line counts and profile rows.
 
 Repeated rollouts are the main profiling lever. `num_repeats=1` is valid, but per-task averages and variance are only meaningful with multiple rollouts per task.
@@ -48,6 +48,6 @@ Load references only when the user needs that detail:
 
 ## Practical Defaults
 
-- Treat `ng_reward_profile` as the reward profiling step; rollout collection does not write reward profile files.
+- Treat `gym eval profile` as the reward profiling step; rollout collection does not write reward profile files.
 - Run strict profiling by default. If rollout collection stopped early, use `++allow_partial_rollouts=True` to profile completed rollouts and drop original input rows with no completed rollout.
 - Trust the target checkout's CLI help and `nemo_gym/reward_profile.py` over memory if flags differ.

.codex/skills/nemo-gym-reward-profiling/references/output-format.md

@@ -6,7 +6,7 @@ Reward profiling is built around joining materialized inputs with rollout result
 
 - `*_materialized_inputs.jsonl`: expanded inputs after repeat expansion. Each row should have `_ng_task_index` and `_ng_rollout_index`.
 - `rollouts.jsonl`: completed rollout results. Each row should have matching `_ng_task_index` and `_ng_rollout_index`.
-- `*_reward_profiling.jsonl`: task-level summaries produced by `ng_reward_profile`.
+- `*_reward_profiling.jsonl`: task-level summaries produced by `gym eval profile`.
 - `*_agent_metrics.json`: agent/global aggregate metrics.
 
 ## Task Profile Rows
@@ -36,7 +36,7 @@ Full model responses stay in `rollouts.jsonl`. Join back to full rows with `(_ng
 
 ## Partial Profiles
 
-Strict profiling is the default. If materialized inputs and rollout results do not have the same rollout keys, `ng_reward_profile` fails and suggests:
+Strict profiling is the default. If materialized inputs and rollout results do not have the same rollout keys, `gym eval profile` fails and suggests:
 
 ```bash
 ++allow_partial_rollouts=True