feat: agent skill evaluation infrastructure

[cwing-nvidia]

Implements #1256 (skill evaluation infrastructure), part of epic #1494.

Summary

Adds skills as a run-level evaluation variable, decoupled from the dataset. Skills are specified via skills.path on gym eval run and applied to a fixed, skill-agnostic dataset at rollout time — no dataset-row coupling (deliberately not modeled like agent_ref). Each rollout result is stamped with a skills_ref for provenance and variant comparison.

Core (agent-agnostic)

• nemo_gym/skills.py — load_skill_directory (parses SKILL.md frontmatter, actionable errors for malformed skills), hash_skill_dir (short content sha256), stage_skills, and SkillsConfig/SkillsRef models.
• rollout_collection.py — optional skills config; resolve once at startup, stamp skills_ref (path + hash + metadata) onto each row, propagate to results.
• skills_ref carries a content hash so optimizer loops (e.g. ACE, GEPA, EvoSkill) that mutate a skill in place at the same path stay distinguishable in reward profiling. Identity is derived from bytes on disk — no optimizer cooperation required.

Claude Code agent (thin per-agent adapter)

• Reads skills_ref.path from the /run request, stages it into the per-request CLAUDE_CONFIG_DIR/skills/, and forces --bare off so native discovery loads it. No skill-interpretation logic added — Claude loads/activates natively.

Docs

• agent-server/agent-skills reference page: workflow, skills_ref hash rationale, and the three-step per-agent adapter contract for future agents (Codex, MCP).

Design notes

The decoupled (run-level, not dataset-row) design and the content-hash variant identity are discussed in #1256 (comment). A committed teaching example + end-to-end "evaluate and optimize skills" tutorial (with a composite-reward example) is tracked separately in #1604.

Test plan

• Unit tests: tests/unit_tests/test_skills.py (incl. spec-compliant metadata.version, UTF-8 BOM tolerance), rollout skills_ref stamping + resume_from_cache round-trip, and agent-side _build_command skills/--bare handling, per-request config-dir cleanup on staging failure, and run() → _create_response skills_path forwarding. ruff clean.
• End-to-end smoke test (reproducible steps below).

Smoke test (reproducible)

Validates both halves end to end against the existing reasoning_gym + Claude Code agent setup, using two variants of the same skill with different content (so hashes differ) and a distinct output marker per variant (to prove activation).

1. Create two throwaway skill variants

mkdir -p /tmp/ng_skill_smoke/variant_a/logic_puzzle_solver /tmp/ng_skill_smoke/variant_b/logic_puzzle_solver

/tmp/ng_skill_smoke/variant_a/logic_puzzle_solver/SKILL.md:

---
name: logic_puzzle_solver
description: Use when solving knights-and-knaves logic puzzles where some characters always tell the truth and others always lie.
---

# Logic Puzzle Solver

When solving a knights-and-knaves puzzle:

1. List each character and the two hypotheses (knight = truth-teller, knave = liar).
2. For each statement, work out what it implies under each hypothesis.
3. Eliminate contradictions until a consistent assignment remains.

Always finish your final answer with the marker: [solved-with-skill-v1]

/tmp/ng_skill_smoke/variant_b/logic_puzzle_solver/SKILL.md — identical except a truth-table approach in the body and the marker [solved-with-skill-v2] (the differing content is what makes the hashes differ).

2. Start the servers (with this branch's code)

gym env start "+config_paths=[resources_servers/reasoning_gym/configs/reasoning_gym_claude_code_agent.yaml]"

3. Collect rollouts for each variant over the same dataset

gym eval run --no-serve \
    +agent_name=reasoning_gym_claude_code_agent \
    +input_jsonl_fpath=resources_servers/reasoning_gym/data/example.jsonl \
    +output_jsonl_fpath=/tmp/skills_smoke_a.jsonl \
    +skills.path=/tmp/ng_skill_smoke/variant_a \
    +limit=1

gym eval run --no-serve \
    +agent_name=reasoning_gym_claude_code_agent \
    +input_jsonl_fpath=resources_servers/reasoning_gym/data/example.jsonl \
    +output_jsonl_fpath=/tmp/skills_smoke_b.jsonl \
    +skills.path=/tmp/ng_skill_smoke/variant_b \
    +limit=1

4. Verify

python -c "import json;print(json.loads(open('/tmp/skills_smoke_a.jsonl').readline())['skills_ref'])"
python -c "import json;print(json.loads(open('/tmp/skills_smoke_b.jsonl').readline())['skills_ref'])"
grep -o 'solved-with-skill-v[12]' /tmp/skills_smoke_*.jsonl

Observed results

	variant A	variant B
skills_ref.path	/tmp/ng_skill_smoke/variant_a	/tmp/ng_skill_smoke/variant_b
skills_ref.hash	4cff9c793395	100c63f5ea97
skills_ref.skills	logic_puzzle_solver (name + description parsed from frontmatter)	same name/description, different content
output marker	solved-with-skill-v1	solved-with-skill-v2

Confirms:

• Gym side: skills_ref stamped + propagated to results, with distinct content hashes for same-named variants.
• Agent side: each variant was staged into its own per-request CLAUDE_CONFIG_DIR/skills/, --bare was dropped, and Claude loaded and followed the correct variant with no cross-contamination (A→v1, B→v2).

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[ritaneves]

For my understanding, this is the backbone needed to evaluate skills on model + benchmark?
If so, can we add features to add cli comparison options (e.g. run model & bench with skills A or B and compare)?

[ffrujeri]

Thanks for this — it's a clean, well-scoped implementation of #1256. The run-level skills.path design (decoupled from the dataset, with a content-hashed skills_ref for variant identity) is sound, and the smoke test plus unit coverage are appreciated. The core mechanics all check out: the skills_ref side-channel into the agent's /run handler, propagation through num_repeats/deepcopy/resume, path resolution, and the async/HTTP/subprocess handling. skills.py is at 100% line coverage and the suite is green.

The inline comments are polish rather than correctness blockers, in three groups:

• Skill loading robustness — parse_skill_md rejects a SKILL.md saved with a UTF-8 BOM (a one-line utf-8-sig fix), and SkillMetadata.version reads a top-level version: key while the Agent Skills spec nests it under metadata:, so the field is effectively dead for spec-compliant skills.
• Robustness / cleanup — the per-run config dir is created outside the try/finally, so a bad skills.path leaks the directory on the error path; and a small doc note that staging skills broadens the runtime (dropping --bare re-enables hooks, plugins, MCP, memory, and project instructions, not just skills) would help anyone trying to measure a skill's isolated impact.
• Test coverage — two suggested additions: an end-to-end test that the skills_ref → metadata side-channel is threaded through /run (the most refactor-fragile line), and a round-trip test that skills_ref survives resume_from_cache.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-cwing-agent-skills-evaluation.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/agent-server/agent-skills

[cwing-nvidia]

@ffrujeri Re-validated end to end after rebasing onto main (now incorporating #1682), where run() forwards skills_path directly to _create_response rather than via the old /v1/responses metadata side-channel.

[cwing-nvidia]

For my understanding, this is the backbone needed to evaluate skills on model + benchmark?
If so, can we add features to add cli comparison options (e.g. run model & bench with skills A or B and compare)?

@ritaneves exactly, this allows you to isolate the impact of skills during evaluation.

Regarding A vs B comparison you can compare by running the same dataset twice with different skills.path and then grouping on skills_ref in gym eval aggregate. We don't have a single-command A/B comparison for any evaluation knob today (e.g. skills, prompts, models, harness etc). eval run always collects one configuration at a time. So this is a generic need rather than something that is skills specific. I opened #1747 to track