feat: Add agent skills for NeMo Gym

[lbliii]

Internal validation only. This PR exercises NeMo Gym end-to-end on a non-trivial workload (paired-rollout grading of agent skills) and ships the harness so other teams can reproduce, validate, and reuse the components. The substantive material — methodology, results, gotchas, upstream pitches — lives in notes/skill-eval/, not in docs/.

Why this exists

Two questions, two days:

1. Can we measure whether a .claude/skills/*/SKILL.md actually helps an agent do its job? Answer: yes, with a paired-rollout 2×2 over (skill-in-prompt × references-on-disk) — and we found one skill that competes with its own references and three skills whose apparent value was mostly the references' value, not the skill's.
2. Is NeMo Gym the right substrate for that kind of evaluation? Answer: yes for skill eval today (rollout collection + service orchestration earned their keep); the methodology ports outside NeMo Gym for generic doc eval. Concrete sharp edges and upstream candidates listed below.

For internal validators: clone, bring it up (5 commands), run a 480-rollout scoreboard (~30 min), and read the results doc. You'll have a working harness and a decision-shaped artifact at the end.

What's here

Three NeMo Gym servers

flowchart LR
  J[input JSONL] --> A[skill_eval_agent</br>/run]
  A -->|/seed_session</br>/run_bash</br>/read_file</br>/close| W[skill_workspace</br>per-session sandbox]
  A -->|/v1/responses| M[policy_model]
  A -->|/verify| JG[skill_judge</br>LLM-as-judge]
  JG -->|/v1/responses| M
  JG --> A
  A --> R[output JSONL</br>reward + per-assertion grades]

Loading

• resources_servers/skill_workspace/ — per-session tmpdir with run_bash + read_file tools. Seeds the skill's scripts/ and references/ based on per-request flags. Sandbox env strips host PATH; workspace-local python → python3 symlink covers macOS rollouts. SKILL.md is never seeded (see contamination note in results.md).
• resources_servers/skill_judge/ — LLM-as-judge. One call grades (response, tool_calls, assertions[]) → grades[] with per-assertion binary verdicts plus evidence strings. Reward = fraction satisfied.
• responses_api_agents/skill_eval_agent/ — orchestrator. Seeds workspace → model↔tool loop → forwards transcript to judge → closes workspace (in finally).

The 2×2 control structure

Two independent flags in verifier_metadata produce four cells per scenario:

	with_references=False	with_references=True
with_skill=False	blind — model priors only	docs-only — realistic reader without the skill pack
with_skill=True	skill-only — SKILL.md in prompt, nothing on disk	skill+docs — realistic reader with the skill pack

The diff tool reports four named marginal effects on three axes (Δreward, Δtools, Δtokens):

• skill | refs=T = skill+docs − docs-only — realistic-deployment value of the skill overlay (the number that matters for shipping).
• skill | refs=F = skill-only − blind — skill as a standalone doc.
• refs | skill=T = skill+docs − skill-only — do refs still matter when the skill is prompted?
• refs | skill=F = docs-only − blind — marginal value of references alone.

Five-field provenance

Every record carries skill_md_sha, evals_sha, fixtures_sha, judge_prompt_sha, harness_version in verifier_metadata. The diff tool tags each delta-of-delta with what changed: md, evals, md+evals, harness, or same-all (no input changed → it's noise/judge drift, not a skill effect).

Tooling

• scripts/build_skill_eval_jsonl.py — emits 4-cell 2×2 with full provenance.
• scripts/diff_skill_scoreboards.py — auto-detects 2×2 vs legacy; renders multi-axis deltas with provenance attribution.

Validate locally (5 commands, ~30 min)

# 1. Setup
uv venv && uv sync --extra dev --group docs

# 2. Configure model endpoint
cat > env.yaml <<'EOF'
policy_base_url: <your /v1 endpoint>
policy_api_key: <your key>
policy_model_name: <your model>
EOF

# 3. Bring up the four servers
ng_run "+config_paths=[
  resources_servers/skill_workspace/configs/skill_workspace.yaml,
  resources_servers/skill_judge/configs/skill_judge.yaml,
  responses_api_models/openai_model/configs/openai_model.yaml,
  responses_api_agents/skill_eval_agent/configs/skill_eval_agent.yaml
]" +skip_venv_if_present=true
ng_status   # expect "4 healthy"

# 4. Build input JSONL (96 records: 4 cells × 8 skills × 3 scenarios)
python scripts/build_skill_eval_jsonl.py \
  --skills-dir .claude/skills \
  --output responses_api_agents/skill_eval_agent/data/example.jsonl

# 5. Collect rollouts + render scoreboard
ng_collect_rollouts \
  +agent_name=skill_eval_agent \
  +input_jsonl_fpath=responses_api_agents/skill_eval_agent/data/example.jsonl \
  +output_jsonl_fpath=results/v8.jsonl \
  +num_repeats=5 +num_samples_in_parallel=6 \
  "+responses_create_params={max_output_tokens: 8192}"
python scripts/diff_skill_scoreboards.py results/v8.jsonl

Pace observed: ~30 min wall-clock for 480 rollouts at 6-way parallel on the NVIDIA inference API. Lighter validation: pass --cells=blind,skill+docs to the build script (only 2 cells × 5 = 240 rollouts, ~15 min).

Headline results (v8, n=5, 480 rollouts)

Realistic-deployment column only. Bold = effect outside ~±0.10 noise floor. Full table + per-skill verdicts in notes/skill-eval/results.md.

skill	skill | refs=T Δreward	Δtools	verdict
gym-run	+0.380	−1.67	load-bearing keeper
add-benchmark	+0.162	−1.13	keep
gym-debug	+0.133	−4.13	keep; biggest efficiency teach
gym-data	+0.093	−3.00	now measurable after scenario rewrite
gym-scaffold-agent	+0.053	−1.00	content gap (non-RL agent patterns missing)
gym-review	+0.048	−4.73	redundant with refs on accuracy; teaches efficiency
gym-profile	+0.040	−2.00	flipped from −0.107 after patterns-table rewrite
gym-config	+0.027	−1.60	2/3 scenarios ceiling-clipped

Three predicted prescriptions, all landed with clean per-skill provenance attribution:

• gym-profile patterns-table rewrite → skill | refs=T flipped −0.107 → +0.040 (Δ +0.147)
• gym-data adversarial scenarios → docs-only 0.97 → 0.88, opening measurable headroom
• gym-review SKILL.md shrink (110 → 53 lines) → realistic preserved, standalone collapsed (context-dependent win)

One universal pattern: every skill reduces tool calls in the realistic cell (−1.00 to −4.73 per rollout). Reward-only scoring is lossy; always read both axes.

NeMo Gym sharp edges (8 items, ~1 hour each the first time)

These are concrete, reproducible, and cheap to fix. Most useful PR output for the framework team:

1. ng_run runs python app.py not python -m. Relative imports break — must use absolute imports from project root.
2. Trailing slashes in policy_base_url produce double-slash 404s on some providers.
3. Some /v1/responses providers return object: "chat.completion"; NeMoGymResponse validation fails on the literal. Normalize in the model server.
4. OpenAI's FunctionToolParam requires explicit strict: False or validation fails at the model server.
5. Host NeMo Gym .venv/bin leaks into subprocess PATH via inherited env; rollouts can see ng_* binaries, Ray sockets, HF/MLflow credentials.
6. Sandbox PATH strip breaks python on macOS — needed a workspace-local python → python3 symlink.
7. Rollout JSONL only persists the final turn's token usage. Multi-turn loops lose intermediate tokens.
8. Workspace cleanup must be inside a finally block — anything else leaks tmpdirs.

Methodology lessons that likely generalize past skill eval:

9. Every artifact seeded into a workspace contaminates the control arm. SKILL.md peek 100% pre-fix; references peek 100% pre-second-fix.
10. Content-hash provenance is necessary but not sufficient. Same-sha runs at temperature=0 still drift up to 0.20 per cell from judge non-determinism.

What's worth absorbing upstream

Two with existing demand. Two parked.

Worth a PR:

• Assertion-grade LLM-as-judge as a shared base class. Three existing in-tree implementations (ours, code_gen, equivalence_llm_judge). Common shape; small base class saves the next team re-inventing it.
• Per-turn token aggregation in the agent base class. Today only the final response's usage reaches the output JSONL; every multi-turn agent eventually wants the sum.

Parked (no second customer yet):

• Paired-arm pattern as a framework primitive (a pairing_key + post-hoc join CLI).
• Framework-level provenance stamping (a narrow ng_stamp_provenance helper that hashes Hydra-config-referenced files).

Read more

• notes/skill-eval/harness.md — build the harness, the 2×2, provenance, iteration loop, sandbox details, scenarios how-to.
• notes/skill-eval/results.md — full v8 scoreboard, per-skill audit (verdicts + before/after edits + v8 outcomes), the contamination story, retraction log, open methodology questions.

Test plan

• 221 core unit tests pass at 96.41% coverage (well above the 96.0% threshold).
• Per-server tests pass: skill_workspace (28), skill_judge (22), skill_eval_agent (9).
• Builder + script tests pass: test_build_skill_eval_jsonl.py (15), test_eval_skills.py (33).
• CI lint clean: ruff format --check, ruff check, README env-list row, secrets-detector hook.
• CI docs build clean: sphinx-build --fail-on-warning succeeds with the skill-eval material moved out of the Sphinx tree.
• End-to-end: ran 480-rollout v8 scoreboard against NVIDIA inference API; provenance tags match expected md / evals+fx / same-all per skill edited.

🤖 Generated with Claude Code