Refactor evaluation skill: vLLM cross-check, MLflow defaults, walltime cap (#1561)

### What does this PR do?

Type of change: documentation

Refactor of the `evaluation` skill (`.claude/skills/evaluation/`) with
several substantive rule additions plus a significant compression pass.

**Skill rules added / tightened:**

- **Cross-check `recipes.vllm.ai` + HF model card before composing the
vLLM command.** Both sources matter; conflicts get surfaced to the user
instead of silently picked. WebFetch caveat triage (3 cases) for
JS-rendered variant tabs (this caveat bit the agent multiple times
during testing; the triage rules name the failure modes).
- **Single `deployment.command:` field** replaces separate
`tensor_parallel_size` / `data_parallel_size` / `extra_args` YAML
fields. NEL mounts the model at `/checkpoint`; Hydra interpolates
`${deployment.port}`.
- **vLLM defaults always included unless a recipe contradicts them**
(silence ≠ contradiction):
  - `--max-num-batched-tokens 8192`
  - `--enable-chunked-prefill`
- `--enable-expert-parallel` (MoE-only, detected via active-param suffix
or `num_experts`-like config field)
- `--max-num-seqs N` where `N = ceil(max_parallelism /
data_parallel_size)`, computed after Step 4 fills in `parallelism`.
- **Six-field evaluation params template:** `parallelism`,
`request_timeout`, `max_retries`, `max_new_tokens`, `temperature`,
`top_p`. No `top_k` / `presence_penalty` / `repetition_penalty` /
`min_p` at top level (task harnesses have their own defaults that
conflict). No per-task `max_new_tokens` overrides — one ceiling
everywhere.
- **`max_new_tokens` mandatory model-card lookup:** highest
card-recommended value wins; "card not yet checked + use generic
default" is explicitly forbidden (this was a real bug the user caught).
- **AA Index v2 (`recipes/tasks/aa/`)** is the default benchmark set for
quantized-checkpoint validation. "AA" / "Artificial Analysis" triggers
AA-only mode (no MMLU-Pro / AIME / LiveCodeBench unless asked).
- **MLflow auto-export on by default in shortcut path**, with
Hydra-interpolated `experiment_name` (`${USER}/${served_model_name}`),
`description` (embeds T / top_p / max_new_tokens), and `tags`
(string-coerced via single quotes for MLflow's tag type requirement).
Only `tracking_uri` needs user input.
- **Walltime capped at 4h** in generated configs (longer walltimes lower
scheduler priority → longer queue). Skill suggests three alternatives
for over-4h runs.
- **Example template** updated to the new conventions; `--max-model-len`
fallback bumped 32K → 131K to cover AA-LCR.
- **`tau2_bench_telecom` recipe:** `parallelism` left as `???` with
rate-limit guidance (canary 32–128, cap 512).
- **`model-card-research.md`:** output-length extraction is now a
mandatory, top-level checklist item with a cross-reference to the
SKILL.md rule.
- **`env.example`:** added `JUDGE_API_KEY` entry for AIME.

**Compression pass:** SKILL.md compressed ~670 → ~330 lines while
preserving every rule. Long prose collapsed into tables and bullets;
duplicated workflow checklist at the bottom removed.

### Usage

The skill is invoked when users ask to evaluate a model. The shortcut
path now produces a config like:

```yaml
deployment:
  command: >-
    vllm serve /checkpoint
    --host 0.0.0.0
    --port ${deployment.port}
    --tensor-parallel-size 8
    ...

evaluation:
  nemo_evaluator_config:
    config:
      params:
        parallelism: ???    # Required — ask user
        request_timeout: 3600
        max_retries: 10
        max_new_tokens: 81920   # from model card (highest)
        temperature: 1.0
        top_p: 0.95

export:
  mlflow:
    tracking_uri: ???
    experiment_name: ${oc.env:USER}/${deployment.served_model_name}
    description: '...'
    tags:
      framework: vllm
      ...
```

### Testing

Manually exercised the skill end-to-end on four models during refactor
(Qwen3.5-122B-A10B-FP8, Qwen3.6-35B-A3B-FP8, Kimi-K2.6, GLM-5.1-NVFP4)
to validate that the cross-check rules surface conflicts, the MLflow
defaults populate correctly, the AA suite excludes the right tasks, and
the walltime cap holds. Test configs are not included in this PR
(session artifacts at repo root, gitignored locally).

### Before your PR is "*Ready for review*"

- Is this change backward compatible?: ✅ — the skill is
editorial/operational; no runtime API changes. Existing configs continue
to work.
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: N/A — skill documentation;
manually exercised on four models.
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A — skill documentation only.
- Did you get Claude approval on this PR?: ❌ — happy to run `/claude
review` if maintainers want it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
- Rewrote the evaluation workflow with a stricter end-to-end checklist,
workspace-reuse guidance, AA-only shortcut, mandatory validation steps,
iterative finalization loop, and a hard 04:00:00 walltime cap
- Enforced vLLM as a single deployment command and stricter
generation/config constraints, including exact top-level params and
model-card–derived max_new_tokens
- Updated env var guidance (JUDGE_API_KEY / INFERENCE_API_KEY), MLflow
export metadata, and registry/auth preflight flow
- Added several AA-task recipes, added new benchmark recipes, and
removed or consolidated legacy task docs

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/NVIDIA/Model-Optimizer/pull/1561?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Author: cjluo-nv

.claude/skills/evaluation/SKILL.md

@@ -18,11 +18,19 @@ NEMO_EVALUATOR_TRUST_PRE_CMD=1
 
 # --- Optional: task-specific keys ---
 
-# HLE, AA-LCR, and other judge-backed tasks
+# Judge / inference endpoints — two separate env vars by harness:
+#
+#   JUDGE_API_KEY     — used by simple-evals harness tasks (e.g. AIME 2025).
+#                       Typically the API key from build.nvidia.com.
+#   INFERENCE_API_KEY — used by nemo-skills and tau2-bench harnesses for
+#                       judge / user-simulator endpoints (HLE, AA-LCR,
+#                       Tau2-Bench Telecom, etc.).
+#
+# The two keys can point to the same provider/credential — they're separate
+# env vars only because different eval harnesses look up different names.
+# Set both if you run tasks from both harness families.
 # JUDGE_API_KEY=
-
-# tau2_bench_telecom user simulator endpoint
-# USER_API_KEY=
+# INFERENCE_API_KEY=
 
 # terminal-bench-hard (AWS sandbox)
 # AWS_ACCESS_KEY_ID=

.claude/skills/evaluation/recipes/env.example

@@ -1,15 +1,13 @@
-# Example: Quantization Validation Suite
+# Example: AA-aligned eval template (single-task starting point)
 #
-# A balanced set of benchmarks for validating quantized model quality.
-# Copy this file and customize for your needs.
-# Task references in recipes/tasks/ define benchmark requirements and YAML
-# fragments — the agent composes them into a runnable config like this one.
+# Minimal config to validate a quantized model end-to-end against one AA-style
+# benchmark. The agent extends this template by copying additional task
+# fragments from recipes/tasks/aa/*.md into the `evaluation.tasks` list — task
+# references define benchmark requirements and YAML fragments, this file
+# defines the deployment + operational conventions.
 #
-# Includes:
-#   - MMLU-Pro (knowledge, completions)
-#   - GPQA Diamond (reasoning, chat, 32 repeats)
-#   - LiveCodeBench v6 (code, chat, 3 repeats)
-#   - IFBench (instruction following, chat, 8 repeats)
+# Includes (default): gpqa_diamond_aa_v3 (simple-evals harness, n_samples=16).
+# Add more AA tasks per recipes/tasks/aa/ and the AA suite rule in SKILL.md.
 #
 # Usage:
 #   nel run --config recipes/examples/example_eval.yaml \
@@ -22,10 +20,15 @@
 # For quantized checkpoints, do not add a vLLM quantization flag by default.
 # Recent vLLM reads ModelOpt quantization metadata from the checkpoint. Only add
 # an explicit flag if the model card, vLLM version, or dry-run error requires it.
-#   -o 'deployment.extra_args=--max-model-len 32768 --trust-remote-code'
+#
+# Deployment uses a single `command:` field instead of separate
+# `tensor_parallel_size` / `data_parallel_size` / `extra_args` fields — the full
+# `vllm serve` invocation lives in the command string. NEL mounts the resolved
+# model (from checkpoint_path or hf_model_handle) at /checkpoint inside the
+# container, and Hydra interpolates ${deployment.port} at run time.
 #
 # Run a single task:
-#   nel run --config ... -t ns_gpqa
+#   nel run --config ... -t gpqa_diamond_aa_v3
 #
 # Canary (2 samples): use this before a full run to validate logs and tune
 # parallelism.
@@ -42,84 +45,68 @@ execution:
   walltime: "04:00:00"
   mounts:
     mount_home: false
+  auto_export:
+    destinations:
+      - mlflow
 deployment:
   env_vars:
     HF_TOKEN: host:HF_TOKEN
   checkpoint_path: ???
   hf_model_handle:
   served_model_name: ???
-  tensor_parallel_size: 1
-  data_parallel_size: 1
-  # For models with custom code, add: --trust-remote-code
-  extra_args: --max-model-len 32768
+  image: vllm/vllm-openai:v0.19.1
+  # For MoE models, add `--enable-expert-parallel` to the command.
+  # For models with custom code, add `--trust-remote-code` to the command.
+  # After filling in evaluation `parallelism` values (top-level + per-task),
+  # append `--max-num-seqs N` to the command where
+  # N = ceil(max_parallelism / data_parallel_size).
+  command: >-
+    vllm serve /checkpoint
+    --host 0.0.0.0
+    --port ${deployment.port}
+    --tensor-parallel-size 1
+    --data-parallel-size 1
+    --max-model-len 131072
+    --max-num-batched-tokens 8192
+    --enable-chunked-prefill
 evaluation:
   env_vars:
     HF_TOKEN: host:HF_TOKEN
   nemo_evaluator_config:
     config:
       params:
+        parallelism: ???  # Number of concurrent requests per each benchmark
         request_timeout: 3600
         max_retries: 10
-        parallelism: 16
+        max_new_tokens: 65536  # 64K for reasoning models; use 16384 (16K) for non-reasoning; prefer model card value
+        temperature: 1.0       # from model card (reasoning mode); adjust per card
+        top_p: 0.95            # from model card (reasoning mode); adjust per card
     target:
       api_endpoint:
         api_key_name: DUMMY_API_KEY
   tasks:
-  # Knowledge (chat endpoint, short)
-    - name: nemo_skills.ns_mmlu_pro
-      nemo_evaluator_config:
-        config:
-          params:
-            extra:
-              num_repeats: 1
-              args: ++prompt_config=eval/aai/mcq-10choices-boxed ++inference.tokens_to_generate=null
-        target:
-          api_endpoint:
-            adapter_config:
-              params_to_remove:
-                - max_new_tokens
-                - max_completion_tokens
-
-  # Reasoning (chat endpoint, 32 repeats, short)
-    - name: ns_gpqa
+  # Reasoning (chat endpoint, 8 repeats, short)
+    - name: gpqa_diamond_aa_v3
+      container: nvcr.io/nvidia/eval-factory/simple-evals:26.03
       nemo_evaluator_config:
         config:
           params:
             extra:
-              args: ++prompt_config=eval/aai/mcq-4choices
-              num_repeats: 32
-        target:
-          api_endpoint:
-            adapter_config:
-              params_to_remove:
-                - max_new_tokens
-                - max_completion_tokens
+              n_samples: 16
 
-  # Code (chat endpoint, 3 repeats, medium)
-    - name: ns_livecodebench
-      nemo_evaluator_config:
-        config:
-          params:
-            extra:
-              dataset_split: test_v6_2408_2505
-              num_repeats: 3
-        target:
-          api_endpoint:
-            adapter_config:
-              params_to_remove:
-                - max_new_tokens
-                - max_completion_tokens
-
-  # Instruction following (chat endpoint, 8 repeats, super short)
-    - name: ns_ifbench
-      nemo_evaluator_config:
-        config:
-          params:
-            extra:
-              num_repeats: 8
-        target:
-          api_endpoint:
-            adapter_config:
-              params_to_remove:
-                - max_new_tokens
-                - max_completion_tokens
+export:
+  mlflow:
+    tracking_uri: ???
+    experiment_name: ${oc.env:USER}/${deployment.served_model_name}
+    description: '${oc.env:USER}/${deployment.served_model_name} | T=${evaluation.nemo_evaluator_config.config.params.temperature}, top_p=${evaluation.nemo_evaluator_config.config.params.top_p},
+      max_new_tokens=${evaluation.nemo_evaluator_config.config.params.max_new_tokens}'
+    log_logs: true
+    log_artifacts: true
+    only_required: false
+    skip_existing: false
+    tags:
+      framework: vllm
+      model: ${deployment.served_model_name}
+      temperature: '${evaluation.nemo_evaluator_config.config.params.temperature}'
+      top_p: '${evaluation.nemo_evaluator_config.config.params.top_p}'
+      max_new_tokens: '${evaluation.nemo_evaluator_config.config.params.max_new_tokens}'

.claude/skills/evaluation/recipes/examples/example_eval.yaml

@@ -0,0 +1,27 @@
+# GPQA Diamond
+
+## Task Details
+
+- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/simple_evals.html#simple-evals-gpqa-diamond-aa-v3>
+
+## Params
+
+## YAML Fragment
+
+Use this inside the top-level `evaluation.tasks` list:
+
+```yaml
+- name: gpqa_diamond_aa_v3
+  container: nvcr.io/nvidia/eval-factory/simple-evals:26.03
+  nemo_evaluator_config:
+    config:
+      params:
+        extra:
+          n_samples: 16
+```
+
+## Score Extraction from mlflow
+
+Result (0-100): `gpqa_diamond_score_micro_avg_of_N`
+
+N is the repeat count.  If the repeat count is unknown, use the highest available `avg_of_N`.

.claude/skills/evaluation/recipes/tasks/aa/gpqa_diamond.md

@@ -0,0 +1,33 @@
+# HLE
+
+## Task Details
+
+- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/nemo_skills.html#nemo-skills-ns-hle-aa>
+
+## Params
+
+This is the text-only HLE task with params aligned to Artificial Analysis Index
+v2. HLE is judge-scored and requires judge credentials.
+
+## YAML Fragment
+
+Use this inside the top-level `evaluation.tasks` list:
+
+```yaml
+- name: ns_hle_aa
+  container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03
+  env_vars:
+    INFERENCE_API_KEY: host:INFERENCE_API_KEY
+  nemo_evaluator_config:
+    config:
+      params:
+        extra:
+          judge:
+            model_id: <hle_aa_judge_model_id>
+            url: <openai_compatible_judge_chat_completions_url>
+            api_key: INFERENCE_API_KEY
+```
+
+## Score Extraction from mlflow
+
+Result (0-100): `hle_pass_at_1_judge_correct`

.claude/skills/evaluation/recipes/tasks/aa/hle.md

@@ -0,0 +1,27 @@
+# IFBench
+
+## Task Details
+
+- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/nemo_skills.html#nemo-skills-ns-ifbench>
+
+## Params
+
+## YAML Fragment
+
+Use this inside the top-level `evaluation.tasks` list:
+
+```yaml
+- name: ns_ifbench
+  container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03
+  nemo_evaluator_config:
+    config:
+      params:
+        extra:
+          num_repeats: 5
+```
+
+## Score Extraction from mlflow
+
+Result (0-100): `ifbench_pass_at_1_avg-of-N_prompt_loose_accuracy`
+
+N is the repeat count.  If the repeat count is unknown, use the highest available `avg-of-N`.

.claude/skills/evaluation/recipes/tasks/aa/ifbench.md

@@ -0,0 +1,50 @@
+# LCR
+
+## Task Details
+
+- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/nemo_skills.html#nemo-skills-ns-aa-lcr>
+
+## Params
+
+Recommended judge: use Qwen3 235B as an OpenAI-compatible equality-checker
+judge, and keep the same judge across comparable runs.
+
+AA-LCR needs long context: plan for roughly 120K input tokens plus 16K
+generation tokens. Set deployment `--max-model-len` to at least `131072`, and
+use a larger value when the model supports it.
+
+## YAML Fragment
+
+LCR has a deployment-side requirement (`--max-model-len 131072`) and a task
+block. Per SKILL.md Step 3, the deployment flag must live inside
+`deployment.command:` — not in the deprecated `extra_args` field.
+
+**Deployment requirement:** ensure the `vllm serve ...` invocation in
+`deployment.command` includes `--max-model-len 131072` (or higher).
+
+```yaml
+- name: ns_aa_lcr
+  container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03
+  env_vars:
+    INFERENCE_API_KEY: host:INFERENCE_API_KEY
+  nemo_evaluator_config:
+    target:
+      api_endpoint:
+        adapter_config:
+          use_request_logging: false
+          use_response_logging: false
+    config:
+      params:
+        extra:
+          num_repeats: 16
+          judge:
+            model_id: <qwen3_235b_judge_model_id>
+            url: <openai_compatible_judge_chat_completions_url>
+            api_key: INFERENCE_API_KEY
+```
+
+## Score Extraction from mlflow
+
+Result (0-100): `aalcr_pass_at_1_avg-of-N_judge_correct`
+
+N is the repeat count.  If the repeat count is unknown, use the highest available `avg-of-N`.

.claude/skills/evaluation/recipes/tasks/aa/lcr.md

@@ -0,0 +1,27 @@
+# MMMU-Pro
+
+## Task Details
+
+- Reference: <https://docs.nvidia.com/nemo/evaluator/latest/evaluation/benchmarks/catalog/all/harnesses/nemo_skills.html>
+
+## Params
+
+MMMU-Pro is a multimodal task. Use a multimodal-capable endpoint.
+
+## YAML Fragment
+
+Use this inside the top-level `evaluation.tasks` list:
+
+```yaml
+- name: ns_mmmu_pro
+  container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03
+  nemo_evaluator_config:
+    config:
+      params:
+        extra:
+          num_repeats: 1
+```
+
+## Score Extraction from mlflow
+
+Result (0-100): `mmmu-pro_pass_at_1_symbolic_correct`