feat: agent skill evaluation infrastructure

fern/versions/latest/pages/agent-server/agent-skills.mdx

@@ -0,0 +1,105 @@
+---
+title: "Agent Skills"
+description: "Evaluate agent skills as a run-level variable, decoupled from the dataset"
+position: 2
+---
+
+# Agent Skills
+
+Skills are reusable units of operational knowledge an agent can load at runtime, following the open [Agent Skills standard](https://agentskills.io/specification) used by Claude Code and Codex CLI. A skill is a **directory** containing a `SKILL.md` file (YAML frontmatter + markdown body) plus optional supporting files.
+
+NeMo Gym treats skills as an **environment-level variable you can sweep**, the same way it treats prompts. You point a run at a directory of skills, the agent loads them with its own native mechanism, and each rollout result is tagged so you can compare variants. NeMo Gym does not interpret or activate skills — it sets up the skill environment and measures the outcome.
+
+## Skills are a run-level knob, not a dataset field
+
+Skills are specified on `gym eval run` and applied to a **fixed, skill-agnostic dataset** at rollout time. The dataset stays a description of tasks, so the same dataset is reusable across skill variants and across agents.
+
+## Usage
+
+A single `skills.path` field points at a directory of skill directories (here `--no-serve` collects against servers already started with `gym env start`):
+
+```bash
+gym eval run --no-serve +agent_name=my_agent \
+  +input_jsonl_fpath=data/tasks.jsonl \
+  +output_jsonl_fpath=results/rollouts_variant_a.jsonl \
+  +skills.path=skills/variant_a/
+```
+
+The directory follows the standard layout — one subdirectory per skill, each with a `SKILL.md`:
+
+```
+skills/variant_a/
+├── cot_enhanced/
+│   └── SKILL.md
+├── tool_focused/
+│   ├── SKILL.md
+│   └── references/
+│       └── api_spec.md
+└── baseline/
+    └── SKILL.md
+```
+
+To compare variants, run again with a different `skills.path` over the same dataset. The skills path is resolved like `input_jsonl_fpath` (relative paths check the working directory, then the Gym root). For distributed runs the directory must be on storage accessible to the agent process.
+
+## Output: `skills_ref`
+
+Each rollout result is stamped with a `skills_ref` for provenance and grouping during reward profiling:
+
+```json
+{
+  "reward": 1.0,
+  "skills_ref": {
+    "path": "skills/variant_a/",
+    "hash": "a1b2c3…",
+    "skills": [{"name": "cot_enhanced", "description": "..."}]
+  }
+}
+```
+
+`hash` is a short content digest of the skill directory. It exists so that optimizer loops (e.g. ACE, GEPA, EvoSkill) that mutate a skill **in place** at the same path still produce distinguishable variants — identity is derived from bytes on disk, requiring no cooperation from the optimizer. Identical content yields an identical hash, so re-testing a prior variant automatically pools with its earlier evaluation.
+
+<Tip>
+For concurrent candidate evaluation (population search), write each candidate to its own directory (`skills/cand-0/`, `skills/cand-1/`, …) to avoid a path-reuse read/write race.
+</Tip>
+
+## How skills reach the agent
+
+NeMo Gym handles everything agent-agnostic: it resolves `skills.path`, computes the `skills_ref`, stamps it onto each request, and propagates it to results. The agent's job is only to make the resolved directory discoverable by its native runtime — because *where* a runtime discovers skills is intrinsically agent-specific.
+
+This splits cleanly into **shared core** and a **thin per-agent adapter**:
+
+| Responsibility | Owner |
+|---|---|
+| `skills.path` config, load/validate, content hash, `skills_ref` stamping + propagation | NeMo Gym core (shared) |
+| Read `skills_ref.path` from the request | Per-agent (identical shape) |
+| Stage the directory into the runtime's discovery location | Per-agent (location differs) |
+| Enable native discovery; load/activate skills | The agent's native runtime |
+
+## Adding skills support to an agent
+
+To support skills in a new agent, implement the three-step adapter. NeMo Gym provides the shared utilities in `nemo_gym.skills`.
+
+1. **Read the skills path from the request.** NeMo Gym stamps `skills_ref` (with `path`) onto the `/run` request body. Read `skills_ref["path"]` in your agent's `run()`.
+
+2. **Stage the directory where your runtime discovers skills**, using `nemo_gym.skills.stage_skills(path, dest_dir)`. The destination is agent-specific:
+
+   | Agent | Discovery location |
+   |---|---|
+   | Claude Code | `CLAUDE_CONFIG_DIR/skills/` |
+   | Codex CLI | the Codex config home (e.g. `$CODEX_HOME`) |
+   | MCP-backed agent | configure the MCP skill server to serve from the directory |
+
+   Prefer staging into a **per-request** ephemeral location so concurrent requests with different skills do not contaminate one another, and so nothing leaks between rollouts.
+
+3. **Enable native discovery.** If your runtime disables discovery by default (Claude Code runs `--bare`), turn it on when skills are present. Then the agent's native loader handles selection and activation — NeMo Gym adds no skill-interpretation logic.
+
+   Note that enabling discovery is usually all-or-nothing: with Claude Code, dropping `--bare` so skills load also re-enables *all* other auto-discovery — hooks, plugins, MCP servers, memory, and `CLAUDE.md` — not just skills. If you are measuring a skill's isolated impact, be aware that a baseline (`--bare`, no skills) versus a skills run may differ by more than the skill content alone.
+
+<Note>
+The reference implementation is the [Claude Code agent](https://github.com/NVIDIA-NeMo/Gym/tree/main/responses_api_agents/claude_code_agent). It stages skills into a fresh per-request `CLAUDE_CONFIG_DIR/skills/` and drops `--bare` when skills are active.
+</Note>
+
+## What NeMo Gym does not do
+
+- It does not parse, select, inject, or activate skills — the agent's native runtime does all of that.
+- It does not add skill-loading capabilities to agents that lack them. Injecting skill content into the prompt for such agents is a possible future enhancement, not part of this design.

nemo_gym/global_config.py

@@ -117,6 +117,7 @@
 RESPONSES_CREATE_PARAMS_KEY_NAME = "responses_create_params"
 RESPONSE_KEY_NAME = "response"
 AGENT_REF_KEY_NAME = "agent_ref"
+SKILLS_REF_KEY_NAME = "skills_ref"
 
 POLICY_BASE_URL_KEY_NAME = "policy_base_url"
 POLICY_API_KEY_KEY_NAME = "policy_api_key"  # pragma: allowlist secret

nemo_gym/rollout_collection.py

@@ -38,6 +38,7 @@
     AGENT_REF_KEY_NAME,
     RESPONSES_CREATE_PARAMS_KEY_NAME,
     ROLLOUT_INDEX_KEY_NAME,
+    SKILLS_REF_KEY_NAME,
     TASK_INDEX_KEY_NAME,
     get_wandb_run,
 )
@@ -51,6 +52,7 @@
     raise_for_status,
     set_global_aiohttp_client,
 )
+from nemo_gym.skills import SkillsConfig, load_skill_directory
 
 
 # ---------------------------------------------------------------------------
@@ -198,6 +200,10 @@ class RolloutCollectionConfig(SharedRolloutCollectionConfig):
         default=None,
         description="Path to a prompt YAML file. Builds responses_create_params.input from the template at rollout time. Mutually exclusive with pre-populated responses_create_params.input in the JSONL data.",
     )
+    skills: Optional[SkillsConfig] = Field(
+        default=None,
+        description="Run-level skills config (skills.path). Makes a directory of Agent Skills standard skills available to the agent at rollout time and stamps each result with a skills_ref. Applied to a skill-agnostic dataset; not a dataset-row field.",
+    )
 
     @model_validator(mode="after")
     def _validate_num_repeats(self) -> "RolloutCollectionConfig":
@@ -268,6 +274,17 @@ def _preprocess_rows_from_config(self, config: RolloutCollectionConfig) -> List[
             prompt_cfg = load_prompt_config(config.prompt_config)
             print(f"Using prompt config: {config.prompt_config}")
 
+        # Resolve skills once for the whole run (hash is content-derived, computed at startup).
+        skills_ref_dict = None
+        if config.skills:
+            skills_ref = load_skill_directory(config.skills.path)
+            skills_ref_dict = skills_ref.model_dump()
+            print(
+                f"Using skills from {config.skills.path} "
+                f"(hash={skills_ref.hash}, {len(skills_ref.skills)} skill(s): "
+                f"{', '.join(s.name for s in skills_ref.skills)})"
+            )
+
         _input_path = Path(config.input_jsonl_fpath)
         if not _input_path.is_absolute():
             _cwd_path = Path.cwd() / _input_path
@@ -305,6 +322,11 @@ def _preprocess_rows_from_config(self, config: RolloutCollectionConfig) -> List[
                 row[RESPONSES_CREATE_PARAMS_KEY_NAME] | responses_create_params_overrides
             )
 
+            # Stamp the run-level skills_ref onto the row so it is sent to the agent in the
+            # /run request body and propagated to results. The source dataset stays untouched.
+            if skills_ref_dict is not None:
+                row[SKILLS_REF_KEY_NAME] = skills_ref_dict
+
             # Resolve task index. Honor a caller-provided value when present (e.g. when an
             # upstream slicer has stamped a globally-stable index across chunks so that
             # subsequent /aggregate_metrics groupby unions chunks correctly); otherwise dedupe
@@ -467,6 +489,8 @@ async def run_from_config(self, config: RolloutCollectionConfig) -> Tuple[List[D
             result[TASK_INDEX_KEY_NAME] = row[TASK_INDEX_KEY_NAME]
             result[ROLLOUT_INDEX_KEY_NAME] = row[ROLLOUT_INDEX_KEY_NAME]
             result[AGENT_REF_KEY_NAME] = row[AGENT_REF_KEY_NAME]
+            if SKILLS_REF_KEY_NAME in row:
+                result[SKILLS_REF_KEY_NAME] = row[SKILLS_REF_KEY_NAME]
 
             no_persist = bool(result.get(NG_NO_PERSIST_KEY))
             failure_class = result.get(NG_FAILURE_CLASS_KEY)

nemo_gym/skills.py

@@ -0,0 +1,201 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Agent skills: a directory of skills made available to an agent at rollout time.
+
+Skills follow the open `Agent Skills standard <https://agentskills.io/specification>`_
+used by Codex CLI and Claude Code. A skill is a *directory* containing a ``SKILL.md``
+file (YAML frontmatter + markdown body) plus optional supporting files. The ``skills.path``
+config points at a directory of such skill directories.
+
+Skills are a run-level knob (specified on ``ng_collect_rollouts``), applied to a fixed,
+skill-agnostic dataset -- mirroring how ``prompt.py`` applies a prompt template. They are
+*not* a dataset-row field, so the same dataset is reusable across skill variants. Each
+rollout result is stamped with a ``skills_ref`` for provenance/grouping in reward profiling.
+
+The ``skills_ref`` carries a content ``hash`` (a short sha256 over the skill directory's
+sorted relative paths + file bytes) so that variants that mutate a skill *in place* at the
+same path -- as optimizer loops like ACE, GEPA, and EvoSkill commonly do -- remain
+distinguishable when comparing rollouts. Identity is derived from bytes on disk, so it
+requires no cooperation from the optimizer.
+"""
+
+import hashlib
+import shutil
+from pathlib import Path
+from typing import List, Optional
+
+import yaml
+from pydantic import BaseModel, Field
+
+from nemo_gym import PARENT_DIR
+
+
+SKILL_MD_FILENAME = "SKILL.md"
+# 12 hex chars (48 bits) is plenty to separate the handful of variants in one experiment
+# while staying readable in tables / W&B.
+_HASH_PREFIX_LEN = 12
+
+
+class SkillMetadata(BaseModel):
+    """Metadata parsed from a single skill's ``SKILL.md`` YAML frontmatter."""
+
+    name: str
+    description: Optional[str] = None
+    version: Optional[str] = None
+
+
+class SkillsRef(BaseModel):
+    """Provenance stamp describing the skills made available for a run.
+
+    Stamped onto rollout result rows (not source datasets). ``hash`` is a content
+    digest so two skill *versions* at the same ``path`` do not collide in profiling.
+    """
+
+    path: str
+    hash: str
+    skills: List[SkillMetadata]
+
+
+class SkillsConfig(BaseModel):
+    """Run-level skills config: ``skills.path`` points at a directory of skill directories."""
+
+    path: str = Field(description="Directory of Agent Skills standard skill directories to make available.")
+
+
+def _resolve_skills_path(path: str) -> Path:
+    """Resolve a skills path. Relative paths check cwd first, then the Gym root (PARENT_DIR).
+
+    This matches how ``input_jsonl_fpath`` and ``config_paths`` are resolved.
+    """
+    p = Path(path)
+    if p.is_absolute():
+        return p
+    cwd_path = Path.cwd() / p
+    return cwd_path if cwd_path.exists() else PARENT_DIR / p
+
+
+def hash_skill_dir(root: Path) -> str:
+    """Compute a stable short sha256 over a skill directory's contents.
+
+    Walks files in sorted relative-path order, folding each file's relative path and bytes
+    into the digest. Including the relative path means renaming or adding/removing files also
+    changes the hash -- a skill *is* its file layout, not just ``SKILL.md``'s bytes.
+    """
+    h = hashlib.sha256()
+    for file_path in sorted(p for p in root.rglob("*") if p.is_file()):
+        h.update(file_path.relative_to(root).as_posix().encode())
+        h.update(b"\0")
+        h.update(file_path.read_bytes())
+        h.update(b"\0")
+    return h.hexdigest()[:_HASH_PREFIX_LEN]
+
+
+def parse_skill_md(skill_md_path: Path) -> SkillMetadata:
+    """Parse a ``SKILL.md`` file's YAML frontmatter into ``SkillMetadata``.
+
+    Frontmatter is delimited by lines containing only ``---``. Raises ``ValueError`` with a
+    clear message if the frontmatter is missing, malformed, or lacks a ``name``.
+    """
+    # utf-8-sig transparently strips a leading BOM if present; Claude Code's own loader tolerates
+    # one, so a BOM should not make Gym reject a skill that would otherwise run.
+    text = skill_md_path.read_text(encoding="utf-8-sig", errors="replace")
+    lines = text.splitlines()
+    if not lines or lines[0].strip() != "---":
+        raise ValueError(
+            f"Skill file {skill_md_path} is missing YAML frontmatter. "
+            f"It must start with a '---' line followed by 'name:' and 'description:' fields."
+        )
+
+    closing_idx = None
+    for idx in range(1, len(lines)):
+        if lines[idx].strip() == "---":
+            closing_idx = idx
+            break
+    if closing_idx is None:
+        raise ValueError(
+            f"Skill file {skill_md_path} has an unterminated YAML frontmatter block (no closing '---' line)."
+        )
+
+    frontmatter_text = "\n".join(lines[1:closing_idx])
+    try:
+        data = yaml.safe_load(frontmatter_text) or {}
+    except yaml.YAMLError as e:
+        raise ValueError(f"Skill file {skill_md_path} has malformed YAML frontmatter: {e}") from None
+    if not isinstance(data, dict):
+        raise ValueError(f"Skill file {skill_md_path} frontmatter must be a YAML mapping, got {type(data).__name__}.")
+    if not data.get("name"):
+        raise ValueError(f"Skill file {skill_md_path} frontmatter is missing a required 'name' field.")
+
+    # The Agent Skills spec nests version under an optional `metadata:` map
+    # (e.g. `metadata: {version: "1.0"}`), not as a top-level key.
+    metadata = data.get("metadata")
+    version = str(metadata["version"]) if isinstance(metadata, dict) and metadata.get("version") is not None else None
+    return SkillMetadata(
+        name=str(data["name"]),
+        description=data.get("description"),
+        version=version,
+    )
+
+
+def load_skill_directory(path: str) -> SkillsRef:
+    """Load a directory of skills, returning a ``SkillsRef`` (path, content hash, metadata).
+
+    The directory at ``path`` contains one subdirectory per skill, each with a ``SKILL.md``.
+    Raises ``ValueError`` with an actionable message if the path is missing, is not a
+    directory, contains no skills, or contains a malformed skill.
+    """
+    resolved = _resolve_skills_path(path)
+    if not resolved.exists():
+        raise ValueError(f"Skills path does not exist: {resolved} (from skills.path={path!r}).")
+    if not resolved.is_dir():
+        raise ValueError(
+            f"Skills path must be a directory of skill directories, but {resolved} is a file "
+            f"(from skills.path={path!r})."
+        )
+
+    skill_dirs = sorted(d for d in resolved.iterdir() if d.is_dir())
+    skills: List[SkillMetadata] = []
+    for skill_dir in skill_dirs:
+        skill_md = skill_dir / SKILL_MD_FILENAME
+        if not skill_md.is_file():
+            raise ValueError(
+                f"Skill directory {skill_dir} is missing a {SKILL_MD_FILENAME} file. "
+                f"Each skill must be a directory containing a {SKILL_MD_FILENAME}."
+            )
+        skills.append(parse_skill_md(skill_md))
+
+    if not skills:
+        raise ValueError(
+            f"Skills path {resolved} contains no skills. Expected one or more subdirectories, "
+            f"each containing a {SKILL_MD_FILENAME} file (from skills.path={path!r})."
+        )
+
+    return SkillsRef(path=path, hash=hash_skill_dir(resolved), skills=skills)
+
+
+def stage_skills(path: str, dest_skills_dir: Path) -> None:
+    """Copy the directory of skills at ``path`` into ``dest_skills_dir``.
+
+    Used by agent runtimes to materialize skills into a location their native discovery
+    mechanism scans (e.g. ``<CLAUDE_CONFIG_DIR>/skills/`` for Claude Code). ``dest_skills_dir``
+    must not already exist. Raises ``ValueError`` if the source path is missing or not a directory.
+    """
+    resolved = _resolve_skills_path(path)
+    if not resolved.is_dir():
+        raise ValueError(
+            f"Cannot stage skills: {resolved} is not a directory (from skills path {path!r}). "
+            f"For distributed runs the skills directory must be on storage accessible to the agent."
+        )
+    shutil.copytree(resolved, dest_skills_dir)