Address review comments

Signed-off-by: Kai Xu <kaix@nvidia.com>

Author: kaix-nv

.claude/skills/evaluation/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: evaluation
-description: Evaluate accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Use when user says "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "accuracy drop", "run nel", or needs to measure how quantization affects model quality. Handles model deployment, config generation, and evaluation execution.
+description: Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "accuracy drop", "run nel". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq) or deploying/serving models (use deployment).
 license: Apache-2.0
 # Based on nel-assistant skill from NeMo Evaluator Launcher (commit f1fa073)
 # https://github.com/NVIDIA-NeMo/Evaluator/tree/f1fa073/packages/nemo-evaluator-launcher/.claude/skills/nel-assistant
@@ -21,7 +21,7 @@ If `MODELOPT_WORKSPACE_ROOT` is set, read `skills/common/workspace-management.md
 ```text
 Config Generation Progress:
 - [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT is set)
-- [ ] Step 1: Check if nel is installed
+- [ ] Step 1: Check if nel is installed and if user has existing config
 - [ ] Step 2: Build the base config file
 - [ ] Step 3: Configure model path and parameters
 - [ ] Step 4: Fill in remaining missing values
@@ -31,11 +31,11 @@ Config Generation Progress:
 - [ ] Step 8: Run the evaluation
 ```
 
-**Step 1: Check if nel is installed**
+**Step 1: Check prerequisites**
 
-Test that `nel` is installed with `nel --version`.
+Test that `nel` is installed with `nel --version`. If not, instruct the user to `pip install nemo-evaluator-launcher`.
 
-If not, instruct the user to `pip install nemo-evaluator-launcher`.
+If the user already has a config file (e.g., "run this config", "evaluate with my-config.yaml"), skip to Step 8. Optionally review it for common issues (missing `???` values, quantization flags) before running.
 
 **Step 2: Build the base config file**
 
@@ -76,6 +76,8 @@ Prompt the user with "I'll ask you 5 questions to build the base config we'll ad
 
 DON'T ALLOW FOR ANY OTHER OPTIONS, only the ones listed above under each category (Execution, Deployment, Auto-export, Model type, Benchmarks). YOU HAVE TO GATHER THE ANSWERS for the 5 questions before you can build the base config.
 
+> **Note:** These categories come from NEL's `build-config` CLI. If `nel skills build-config --help` shows different options than listed above, use the CLI's current options instead.
+
 When you have all the answers, run the script to build the base config:
 
 ```bash
@@ -118,14 +120,7 @@ If no `hf_quant_config.json`, also check `config.json` for a `quantization_confi
 
 **Quantization-aware benchmark defaults:**
 
-When a quantized checkpoint is detected, recommend benchmarks sensitive to quantization accuracy loss:
-
-- **Always include**: MMLU (general knowledge — typically shows measurable accuracy loss from quantization)
-- **Recommended**: GSM8K (math reasoning — sensitive to precision loss), ARC-Challenge (reasoning)
-- **Good to add**: HumanEval (code generation — catches subtle degradation), Winogrande (commonsense)
-- **Less useful for quant comparison**: IFEval (instruction following — typically less affected, but worth including for aggressive quantization like FP4)
-
-Present these recommendations to the user and ask which to include. If the user already specified benchmarks, keep their choice but mention any accuracy-sensitive benchmarks they may have missed.
+When a quantized checkpoint is detected, read `references/quantization-benchmarks.md` for benchmark sensitivity rankings and recommended sets. Present recommendations to the user and ask which to include.
 
 Read `references/model-card-research.md` for the full extraction checklist (sampling params, reasoning config, ARM64 compatibility, pre_cmd, etc.). Use WebSearch to research the model card, present findings, and ask the user to confirm.
 
@@ -191,7 +186,7 @@ Print the following commands to the user. Propose to execute them in order to co
 **Important**: Export required environment variables based on your config. If any tokens or keys are missing (e.g. `HF_TOKEN`, `NGC_API_KEY`, `api_key_name` from the config), ask the user to put them in a `.env` file in the project root so you can run `set -a && source .env && set +a` (or equivalent) before executing `nel run` commands.
 
 ```bash
-# If using pre_cmd or post_cmd:
+# If using pre_cmd or post_cmd (review pre_cmd content before enabling — it runs arbitrary commands):
 export NEMO_EVALUATOR_TRUST_PRE_CMD=1
 
 # If using nemo_skills.* tasks with self-deployment:
@@ -299,7 +294,7 @@ Now, copy this checklist and track your progress:
 ```text
 Config Generation Progress:
 - [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT is set)
-- [ ] Step 1: Check if nel is installed
+- [ ] Step 1: Check if nel is installed and if user has existing config
 - [ ] Step 2: Build the base config file
 - [ ] Step 3: Configure model path and parameters
 - [ ] Step 4: Fill in remaining missing values

.claude/skills/evaluation/references/quantization-benchmarks.md

@@ -0,0 +1,26 @@
+# Quantization-Aware Benchmark Recommendations
+
+When evaluating a quantized checkpoint, prioritize benchmarks that are sensitive to precision loss.
+
+## Sensitivity ranking
+
+| Priority | Benchmarks | Why |
+|----------|-----------|-----|
+| **Always include** | MMLU | General knowledge — typically shows measurable accuracy loss from quantization |
+| **Recommended** | GSM8K, ARC-Challenge | Math reasoning and general reasoning — sensitive to precision loss |
+| **Good to add** | HumanEval, Winogrande | Code generation and commonsense — catches subtle degradation |
+| **Less useful for quant comparison** | IFEval | Instruction following — typically less affected, but worth including for aggressive quantization like FP4 |
+
+## Recommended sets by use case
+
+| Use case | Benchmarks |
+|----------|-----------|
+| Quick sanity check | MMLU |
+| Standard quant validation | MMLU, GSM8K, ARC-Challenge |
+| Thorough evaluation | MMLU, GSM8K, ARC-Challenge, HumanEval, Winogrande |
+| Code-focused model | HumanEval, MBPP, MMLU |
+| Reasoning model | GSM8K, MATH-500, GPQA, MMLU |
+
+## How to use
+
+Present these recommendations to the user and ask which to include. If the user already specified benchmarks, keep their choice but mention any accuracy-sensitive benchmarks they may have missed.

.claude/skills/evaluation/tests/evals.json

@@ -0,0 +1,65 @@
+[
+  {
+    "name": "nemotron3-nano-bf16-reasoning",
+    "skills": ["evaluation"],
+    "query": "Help me evaluate Nemotron 3 Nano BF16 from NVIDIA",
+    "files": [],
+    "expected_behavior": [
+      "Verifies nel is installed by running 'nel --version'",
+      "Asks all 5 base config questions (execution, deployment, auto-export, model type, benchmarks) before generating the config",
+      "Runs 'nel skills build-config' with correct flags matching user answers: --execution slurm --deployment vllm --model-type reasoning --benchmarks standard code math_reasoning --export mlflow",
+      "Searches the web for the model card on HuggingFace and extracts model-specific settings",
+      "Sets correct HF handle: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
+      "Sets reasoning sampling params from model card: temperature=1.0, top_p=1.0",
+      "Configures reasoning toggle via params_to_add with chat_template_kwargs.enable_thinking (not via system prompt)",
+      "Disables reasoning for IFEval task using enable_thinking: false with use_system_prompt: false",
+      "Adds deployment.pre_cmd using curl (not wget) to download nano_v3_reasoning_parser.py from HuggingFace",
+      "Adds vLLM extra_args including --trust-remote-code, --reasoning-parser-plugin, --reasoning-parser nano_v3, --max-num-seqs 8",
+      "Pins vLLM image to v0.12.0 or later as required by model card",
+      "Adds target.api_endpoint.api_key_name: DUMMY_API_KEY for nemo_skills tasks with self-deployment",
+      "Fills in all ??? placeholders after asking the user for SLURM hostname, account, output_dir, MLflow tracking_uri, and experiment_name",
+      "Applies user-requested SLURM customizations: partition batch_short, walltime 00:20:00, MLflow tag scenario: demo",
+      "Presents task list and waits for user confirmation before proceeding",
+      "Configures request and response logging interceptors under evaluation.nemo_evaluator_config.config.target.api_endpoint.adapter_config using correct field names (max_logged_requests/max_logged_responses, not max_saved_*)",
+      "Handles dry-run failure for missing HF_TOKEN_FOR_GPQA_DIAMOND by offering to fix the config",
+      "Successfully submits test run with limit_samples=10 after dry-run passes",
+      "Provides monitoring commands (nel status, nel info --logs) and inspects server logs via SSH when asked"
+    ]
+  },
+  {
+    "name": "quantized-checkpoint-local-vllm",
+    "skills": ["evaluation"],
+    "query": "evaluate my FP8 quantized Llama checkpoint at ./llama-3.1-8b-fp8 on MMLU and GSM8K",
+    "files": [],
+    "expected_behavior": [
+      "Verifies nel is installed by running nel --version",
+      "Asks all 5 base config questions (execution, deployment, auto-export, model type, benchmarks)",
+      "Runs nel skills build-config with correct flags matching user answers",
+      "Sets deployment.checkpoint_path to ./llama-3.1-8b-fp8 and deployment.hf_model_handle to null",
+      "Auto-detects quantization format by reading ./llama-3.1-8b-fp8/hf_quant_config.json",
+      "Finds quant_algo=FP8 and adds --quantization modelopt to deployment.extra_args",
+      "Recommends accuracy-sensitive benchmarks from references/quantization-benchmarks.md",
+      "Searches web for Llama-3.1-8B model card and extracts sampling params, context length, TP settings",
+      "Fills in remaining missing values by asking user",
+      "Runs dry-run, then test with limit_samples=10, then full evaluation",
+      "Reports accuracy results per benchmark"
+    ]
+  },
+  {
+    "name": "slurm-quantized-model",
+    "skills": ["evaluation"],
+    "query": "Evaluate my quantized Llama-3.1-8B-FP8 checkpoint on mmlu and gsm8k on the SLURM cluster",
+    "files": [],
+    "expected_behavior": [
+      "Verifies nel is installed by running nel --version",
+      "Asks 5 base config questions with execution=slurm pre-selected based on user request",
+      "Runs nel skills build-config with --execution slurm --deployment vllm --benchmarks standard",
+      "Detects FP8 quantization from hf_quant_config.json and sets deployment.extra_args with --quantization modelopt",
+      "Reads references/quantization-benchmarks.md and recommends accuracy-sensitive benchmarks",
+      "Uses WebSearch to research model card for sampling params and context length",
+      "Fills in SLURM-specific values: hostname, account, partition from user input",
+      "Runs dry-run validation before full evaluation",
+      "Provides SSH-based log monitoring commands for SLURM execution"
+    ]
+  }
+]