Generic

Author: soluwalana

plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/SKILL.md

@@ -113,7 +113,7 @@ Training never runs inside the `nemo` CLI process. After `submit`, the platform'
 - Resolve the CLI per **Pre-flight — CLI resolution** before any `nemo …` command; run from the **nemo-platform** git root, not a plugin subfolder.
 - Set `NEMO_BASE_URL` (or `NMP_BASE_URL`) only when the user gives a platform URL; default `http://127.0.0.1:8080` (same as `http://localhost:8080`). Track whether the user **overrode** the base URL — see **Platform unreachable** below.
 - **Platform unreachable** — if any platform API call fails with a connection error (`Connection error`, timeout, refused):
-  - **User gave a custom URL** (e.g. `10.0.0.51:8080`) or you exported a non-default `NEMO_BASE_URL` / `NMP_BASE_URL`: stop and tell the user the platform is not reachable at that address. Do **not** offer to start local services.
+  - **User gave a custom URL** (e.g. `$NMP_BASE_URL`) or you exported a non-default `NMP_BASE_URL` / `NEMO_BASE_URL`: stop and tell the user the platform is not reachable at that address. Do **not** offer to start local services.
   - **Default URL only** (no user override): **ask** whether to start the platform locally. If they agree, from the **nemo-platform** git root run in the **background**:
 
     ```bash
@@ -139,7 +139,7 @@ Training never runs inside the `nemo` CLI process. After `submit`, the platform'
 - **Do not use local `docker info`** to pick automodel vs unsloth. Run `nemo jobs list-execution-profiles -f json` against the user's platform (login first only if auth is enabled — see **Authentication**; see `references/troubleshooting.md`). Default output is a table — **`-f json` is required** for scripting; parse **stdout only** (do not pipe `2>&1` into `json.load`).
 - **Do not merge stderr into stdout when parsing JSON** — `submit`, `explain`, and `-f json` commands write **JSON on stdout**; harmless warnings like `Configuration file not found, using defaults` go to **stderr**. Piping with **`2>&1`** before `json.load` raises `JSONDecodeError` even when submit **succeeded** — a common cause of **duplicate jobs** when the agent re-submits after a parse error. Parse stdout only; redirect stderr if needed (`2>/dev/null`). See `references/troubleshooting.md` § **Parsing CLI JSON**.
 - For submit/image/plugin errors (both backends), read `references/troubleshooting.md`. Unsloth needs the `nmp-unsloth-training` container image on the **platform host's** Docker daemon (see `docker/unsloth/README.md`).
-- **Missing training image on a remote platform** — if the user gave a non-localhost `NEMO_BASE_URL` / `NMP_BASE_URL` (e.g. `10.0.0.51:8080`) and the job errors with `Failed to pull image`, `manifest unknown`, or missing `nmp-unsloth-training` / automodel training image: **do not** run `docker build`, `docker pull`, or `docker buildx bake` on the agent machine. Report with **Report to user** (use **Output adapter fileset (planned):** on error), then append on-target build steps from `references/troubleshooting.md` § **Missing training images**.
+- **Missing training image on a remote platform** — if the user gave a non-localhost `NMP_BASE_URL` / `NEMO_BASE_URL` and the job errors with `Failed to pull image`, `manifest unknown`, or missing `nmp-unsloth-training` / automodel training image: **do not** run `docker build`, `docker pull`, or `docker buildx bake` on the agent machine. Report with **Report to user** (use **Output adapter fileset (planned):** on error), then append on-target build steps from `references/troubleshooting.md` § **Missing training images**.
 - **Gated HuggingFace models** (Llama, Gemma, …) — confirm `hf-token` + fileset `token_secret` before submit; download fails with `Failed to access upstream storage` / 502 when missing. See **HuggingFace token (gated models)** and `references/troubleshooting.md` § **Gated HuggingFace models**.
 - **Post-training eval format** — use the same CHAT `messages` JSONL as training. **Do not** flatten rows to `prompt`/`expected` for the evaluator. Send `messages[:-1]` at inference (exclude final assistant label); score against `messages[-1].content`. See `references/post-training-eval.md` and `references/eval_helpers.py`.
 - **LoRA adapters load automatically for eval** — when a job completes, the adapter is registered on the model entity and hot-reloaded on any **READY** deployment with `lora_enabled: true`. **Do not** update deployments or providers before eval. **Do** route LoRA eval through the **provider** gateway (`/provider/<name>/-/v1` with `model: default--<adapter>`); the model-entity path (`/model/<entity>/-/v1`) always hits the base model. See `references/post-training-eval.md` § **Request routing (base vs LoRA)**.
@@ -600,8 +600,8 @@ The adapter `<output.name>` is registered on `default/<model-entity>`. Weights a
 
 | Target | Gateway path | OpenAI base URL | Request `"model"` field |
 |--------|--------------|-----------------|-------------------------|
-| **Base** weights | model-entity | `<NEMO_BASE_URL>/apis/inference-gateway/v2/workspaces/default/model/<model-entity>/-/v1` | `default/<model-entity>` |
-| **LoRA adapter** | **provider** | `<NEMO_BASE_URL>/apis/inference-gateway/v2/workspaces/default/provider/<provider>/-/v1` | `default--<output.name>` |
+| **Base** weights | model-entity | `$NMP_BASE_URL/apis/inference-gateway/v2/workspaces/default/model/<model-entity>/-/v1` | `default/<model-entity>` |
+| **LoRA adapter** | **provider** | `$NMP_BASE_URL/apis/inference-gateway/v2/workspaces/default/provider/<provider>/-/v1` | `default--<output.name>` |
 
 **Common mistake:** posting to the model-entity URL with `"model": "default--<output.name>"` still runs the **base** model. Base-vs-adapter eval will look identical until LoRA requests use the **provider** URL above. See `references/post-training-eval.md` § **Request routing (base vs LoRA)**.
 
@@ -619,7 +619,7 @@ Match training context at inference — send **`messages[:-1]`** (all turns exce
 #### Example — LoRA adapter via provider
 
 \`\`\`bash
-export NEMO_BASE_URL=<platform-url>   # omit when using default localhost
+export NMP_BASE_URL=<platform-url>   # omit when using default localhost; NEMO_BASE_URL also works
 nemo inference gateway provider post v1/chat/completions <provider> --workspace default \\
   --body '{
     "model": "default--<output.name>",
@@ -633,7 +633,7 @@ nemo inference gateway provider post v1/chat/completions <provider> --workspace
 #### Example — base model via model-entity (comparison)
 
 \`\`\`bash
-export NEMO_BASE_URL=<platform-url>
+export NMP_BASE_URL=<platform-url>
 nemo inference gateway model post v1/chat/completions <model-entity> --workspace default \\
   --body '{
     "model": "default/<model-entity>",
@@ -651,18 +651,17 @@ Validation loss from training is **not** accuracy. To compare base vs adapter on
 \`\`\`bash
 cd /path/to/nemo-platform
 uv run python plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/eval_helpers.py \\
-  --base-url <platform-url> \\
   --model-entity <model-entity> \\
   --adapter <output.name> \\
   --provider <provider> \\
   --dataset-fileset <dataset-fileset> \\
   --split validation.jsonl
 \`\`\`
 
-Uses CHAT `messages` rows unchanged from the training fileset (`messages[:-1]` at inference). Repeat `--adapter` for multi-adapter compare. `--provider` is optional when a READY provider is auto-discovered.
+Uses CHAT `messages` rows unchanged from the training fileset (`messages[:-1]` at inference). Repeat `--adapter` for multi-adapter compare. `--provider` is optional when a READY provider is auto-discovered. Set `NMP_BASE_URL` (or pass `--base-url`) when the platform is not localhost — the helper reads `$NMP_BASE_URL` / `$NEMO_BASE_URL` by default.
 ```
 
-Use the user's platform URL in `NEMO_BASE_URL` when they overrode it; omit the export line for default `http://127.0.0.1:8080`. Substitute `<provider>`, `<NEMO_BASE_URL>`, and entity/adapter names with values from discovery — do not leave generic placeholders in the user-facing report. Do **not** tell the user to update the deployment or add the adapter to a provider before calling it — registration on the model entity is sufficient.
+Use the user's platform URL in `NMP_BASE_URL` when they overrode it; omit the export line for default `http://127.0.0.1:8080`. Substitute `<provider>`, concrete URLs, and entity/adapter names with values from discovery — do not leave generic placeholders in the user-facing report. Do **not** tell the user to update the deployment or add the adapter to a provider before calling it — registration on the model entity is sufficient.
 
 **Save report to `/tmp`** — unless the user opts out, write the full Markdown report (header, **Training configuration**, **Using the adapter** when `completed`, and **Resources created** when a slug or new filesets were used) to `/tmp/fine-tune-result-<slug-or-job-suffix>.md`. Use the random slug from the run when one was assigned; otherwise use the job id suffix (e.g. `a925b07ff678`).
 

plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/eval_helpers.py

@@ -6,11 +6,13 @@
 LoRA adapters registered on the model entity are hot-reloaded automatically on
 deployments with ``lora_enabled: true`` — no deployment update before eval.
 
-Run from the nemo-platform git root::
+Run from the nemo-platform git root (reads ``$NMP_BASE_URL`` / ``$NEMO_BASE_URL`` when
+``--base-url`` is omitted)::
 
+    export NMP_BASE_URL=http://127.0.0.1:8080
     uv run python plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/eval_helpers.py \\
-        --model-entity qwen3-1.7b --adapter lora-a --adapter lora-b \\
-        --provider qwen3-1.7b-csqa-lora-deploy --dataset-fileset commonsense_qa --split validation
+        --model-entity <model-entity> --adapter <adapter-a> --adapter <adapter-b> \\
+        --provider <provider> --dataset-fileset <dataset-fileset> --split validation.jsonl
 
 Import in agent scripts (add references/ to sys.path or run via uv from repo root).
 """
@@ -19,6 +21,7 @@
 
 import argparse
 import json
+import os
 import re
 import urllib.error
 import urllib.request
@@ -667,9 +670,22 @@ def build_eval_payload(
     return payload
 
 
+def default_base_url() -> str:
+    """Platform URL from env or localhost default."""
+    return (
+        os.environ.get("NMP_BASE_URL")
+        or os.environ.get("NEMO_BASE_URL")
+        or "http://127.0.0.1:8080"
+    )
+
+
 def _parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(description="Compare base vs LoRA on CHAT validation JSONL")
-    parser.add_argument("--base-url", default="http://127.0.0.1:8080")
+    parser.add_argument(
+        "--base-url",
+        default=default_base_url(),
+        help="Platform URL (default: $NMP_BASE_URL, $NEMO_BASE_URL, or http://127.0.0.1:8080)",
+    )
     parser.add_argument("--workspace", default="default")
     parser.add_argument("--model-entity", required=True)
     parser.add_argument(

plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/post-training-eval.md

@@ -9,8 +9,8 @@ Training and evaluation must use the **same CHAT JSONL row shape**:
 ```json
 {
   "messages": [
-    {"role": "user", "content": "Question: …\nChoices:\n…\nAnswer:"},
-    {"role": "assistant", "content": "bank"}
+    {"role": "user", "content": "<user turn or multi-turn prompt>"},
+    {"role": "assistant", "content": "<label to predict>"}
   ]
 }
 ```
@@ -23,7 +23,7 @@ Multi-turn rows use the same rule: the **final** `messages[-1]` turn is the assi
 | Send **`messages[:-1]`** at inference (exclude only the final assistant label) | Pass full `messages` including the label turn, or use `{"messages": "{{ item.messages }}"}` unfiltered |
 | Score against **`messages[-1].content`** (final assistant turn) | Score against a renamed `expected` field unless you also keep `messages` |
 
-Single-turn MCQA (user + assistant) is the degenerate case: `messages[:-1]` is just the user turn.
+Single-turn rows (one user prompt + one assistant label) are the degenerate case: `messages[:-1]` is just the user turn.
 
 Automodel and unsloth both train on this shape when `has_chat` is true (see `hf-conversion.md`, `dataset-formats.md`).
 
@@ -38,7 +38,7 @@ CHAT_REFERENCE_TEMPLATE = "{{ item.messages[-1].content }}"
 
 Import from `references/eval_helpers.py` — do not re-type these in one-off scripts.
 
-## Inference defaults (Qwen3 / thinking models)
+## Inference defaults (thinking models, e.g. Qwen3)
 
 | Setting | Recommended | Avoid |
 |---------|-------------|-------|
@@ -67,8 +67,10 @@ The model-entity proxy path **always** resolves to the base VirtualModel. Settin
 
 | Target | Gateway route | URL pattern | Request `model` field |
 |--------|---------------|-------------|------------------------|
-| Base entity | **Model entity** | `{NEMO_BASE_URL}/apis/inference-gateway/v2/workspaces/default/model/<model-entity>/-/v1` | `default/<model-entity>` |
-| LoRA adapter | **Provider** | `{NEMO_BASE_URL}/apis/inference-gateway/v2/workspaces/default/provider/<provider>/-/v1` | `default--<adapter-name>` |
+| Base entity | **Model entity** | `$NMP_BASE_URL/apis/inference-gateway/v2/workspaces/default/model/<model-entity>/-/v1` | `default/<model-entity>` |
+| LoRA adapter | **Provider** | `$NMP_BASE_URL/apis/inference-gateway/v2/workspaces/default/provider/<provider>/-/v1` | `default--<adapter-name>` |
+
+(`NEMO_BASE_URL` is an alias for `NMP_BASE_URL`.)
 
 `eval_helpers.py` auto-discovers a READY provider that serves the base entity (or pass `--provider <name>`). Adapter weights still hot-reload on the deployment — no provider update per adapter.
 
@@ -83,37 +85,40 @@ Optional sanity checks:
 
 If base and LoRA scores were identical (~99% same outputs), the adapter was almost certainly called through the **model-entity** path. That path always resolves to the base VirtualModel — the `"model": "default--<adapter>"` field in the body is ignored. Fix: route LoRA through the **provider** URL with the same `model` field. `eval_helpers.build_platform_model_target()` and the CLI implement this split automatically.
 
-### MCQA metric interpretation
+### Classification / short-answer metric interpretation
 
-For commonsense_qa-style MCQA, treat **`normalized_accuracy`** as the primary metric (`normalize_mcqa_answer` strips `A. foo` / markdown).
+For multiple-choice or short-label SFT, treat **`normalized_accuracy`** as the primary metric when labels need normalization (`normalize_mcqa_answer` strips `A. foo`, markdown, etc.).
 
 | Observation | Likely meaning |
 |-------------|----------------|
-| Base & LoRA both ~59% normalized, within ~1 pp | LoRA hit **model-entity** path (base only) — check `warnings` and gateway logs |
-| Base raw exact 0%, normalized ~59% | Normal for base on MCQA (formatted prose answers) |
-| LoRA normalized >> base (e.g. 76% vs 59%) | Correct provider routing and real adapter lift |
-| Train loss dropped sharply but eval flat | Wrong eval routing or need more epochs — val loss ≠ accuracy |
+| Base & LoRA normalized scores match within ~1–2 pp | LoRA likely hit **model-entity** path (base only) — check `warnings` and gateway logs |
+| Base raw exact match low, normalized much higher | Normal when the base model emits formatted prose but normalized labels match |
+| LoRA normalized clearly above base | Correct provider routing and real adapter lift |
+| Train loss dropped sharply but eval flat | Wrong eval routing, mismatched inference settings, or need more epochs — val loss ≠ accuracy |
 
 ### Epoch / adapter ablations
 
 Resolve adapter names from completed job specs instead of guessing:
 
 ```python
+import os
 from eval_helpers import list_completed_job_adapters, compare_adapters, build_eval_payload
 
+base_url = os.environ.get("NMP_BASE_URL") or os.environ.get("NEMO_BASE_URL") or "http://127.0.0.1:8080"
+
 jobs = list_completed_job_adapters(
-    base_url="http://10.0.0.51:8080",
+    base_url=base_url,
     workspace="default",
-    model_entity="qwen3-1.7b",
-    dataset_fileset="commonsense_qa",
+    model_entity="<model-entity>",
+    dataset_fileset="<dataset-fileset>",
 )
 # jobs[0].epochs, jobs[0].adapter_name, jobs[0].backend — sorted newest first
 
 summaries = compare_adapters(
-    base_url="...",
+    base_url=base_url,
     workspace="default",
-    model_entity="qwen3-1.7b",
-    adapter_names=[jobs[0].adapter_name, jobs[2].adapter_name],
+    model_entity="<model-entity>",
+    adapter_names=[jobs[0].adapter_name, jobs[1].adapter_name],
     rows=rows,
 )
 payload = build_eval_payload(..., summaries=summaries, adapter_names=[...])
@@ -129,7 +134,7 @@ When comparing adapters from **different backends** (automodel vs unsloth) or ba
 | HTTP base URL | `…/provider/<provider>/-/v1` | `…/model/<model-entity>/-/v1` |
 | `"model"` | `default--<adapter-name>` | `default/<model-entity>` |
 | `messages` | `messages[:-1]` from the training row (exclude final assistant label) | Same |
-| Qwen3 short SFT | `"chat_template_kwargs": {"enable_thinking": false}` | Same |
+| Short-answer SFT (e.g. Qwen3) | `"chat_template_kwargs": {"enable_thinking": false}` | Same |
 | `max_tokens` / `temperature` | `64` / `0` typical for short labels | Same |
 
 CLI shortcuts (substitute names from the job):
@@ -158,23 +163,23 @@ Val loss from training is **not** accuracy — always run a generation eval for
 From **nemo-platform** git root:
 
 ```bash
-# Base vs one adapter
+export NMP_BASE_URL=http://127.0.0.1:8080   # user platform URL when not localhost
+
+# Base vs one adapter (--base-url optional when NMP_BASE_URL is set)
 uv run python plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/eval_helpers.py \
-  --base-url http://10.0.0.51:8080 \
-  --model-entity qwen3-1.7b \
-  --adapter qwen3-1.7b-csqa-unsloth-jun16-e3 \
-  --provider qwen3-1.7b-csqa-lora-deploy \
-  --dataset-fileset commonsense_qa \
+  --model-entity <model-entity> \
+  --adapter <adapter-name> \
+  --provider <provider> \
+  --dataset-fileset <dataset-fileset> \
   --split validation.jsonl \
   --output /tmp/fine-tune-eval.json
 
 # Base vs multiple adapters (epoch ablation)
 uv run python plugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/eval_helpers.py \
-  --base-url http://10.0.0.51:8080 \
-  --model-entity qwen3-1.7b \
-  --adapter qwen3-1.7b-commonsense-qa-lora-jun12-v2 \
-  --adapter qwen3-1.7b-csqa-unsloth-jun16-e3 \
-  --dataset-fileset commonsense_qa \
+  --model-entity <model-entity> \
+  --adapter <adapter-a> \
+  --adapter <adapter-b> \
+  --dataset-fileset <dataset-fileset> \
   --split validation.jsonl \
   --output /tmp/fine-tune-eval-multi.json
 ```