Merge branch 'main' into ajarasane/resnet_support

Author: kevalmorabia97

.claude/skills/common/slurm-setup.md

@@ -192,3 +192,128 @@ chmod -R g+rwX /path/to/.hf_cache/
 ```
 
 Scope `chmod` to only the directories the job needs — avoid world-writable paths on shared clusters.
+
+---
+
+## 6. Container Registry Authentication
+
+**Before submitting any SLURM job that pulls a container image**, check that the cluster has credentials for the image's registry. Missing auth causes jobs to fail after waiting in the queue — a costly mistake.
+
+### Step 1: Detect the container runtime
+
+Different clusters use different container runtimes. Detect which is available:
+
+```bash
+# On the cluster (or via ssh):
+which enroot 2>/dev/null && echo "RUNTIME=enroot"
+which docker 2>/dev/null && echo "RUNTIME=docker"
+```
+
+| Runtime | Typical clusters | SLURM integration |
+| --- | --- | --- |
+| **enroot/pyxis** | NVIDIA internal (DGX Cloud, EOS, Selene, GCP-NRT) | `srun --container-image` |
+| **Docker** | Bare-metal / on-prem with GPU | `docker run` inside job script |
+
+### Step 2: Check credentials for the image's registry
+
+Determine the registry from the image URI:
+
+| Image pattern | Registry |
+| --- | --- |
+| `nvcr.io/nvidia/...` | NGC |
+| `vllm/vllm-openai:...`, `lmsysorg/sglang:...`, or no registry prefix | DockerHub |
+| `ghcr.io/...` | GitHub Container Registry |
+| `docker.io/...` | DockerHub (explicit) |
+
+Then check credentials based on the runtime:
+
+#### enroot/pyxis
+
+```bash
+grep -E '^\s*machine\s+' ~/.config/enroot/.credentials 2>/dev/null
+```
+
+Look for `machine <registry>` lines:
+- NGC → `machine nvcr.io`
+- DockerHub → `machine auth.docker.io`
+- GHCR → `machine ghcr.io`
+
+#### Docker
+
+```bash
+cat ~/.docker/config.json 2>/dev/null | python3 -c "import json,sys; print('\n'.join(json.load(sys.stdin).get('auths', {}).keys()))"
+```
+
+Look for registry keys (`https://index.docker.io/v1/`, `nvcr.io`, `ghcr.io`).
+
+### Step 3: If credentials are missing
+
+**Do not submit the job.** Instead:
+
+1. Tell the user which registry and runtime need authentication
+2. Show the fix for their runtime:
+
+**enroot/pyxis:**
+
+```bash
+mkdir -p ~/.config/enroot
+
+# DockerHub (get token from https://hub.docker.com/settings/security)
+cat >> ~/.config/enroot/.credentials << 'EOF'
+machine auth.docker.io
+  login <dockerhub_username>
+  password <access_token>
+EOF
+
+# NGC (get API key from https://org.ngc.nvidia.com/setup/api-keys)
+cat >> ~/.config/enroot/.credentials << 'EOF'
+machine nvcr.io
+  login $oauthtoken
+  password <ngc_api_key>
+EOF
+```
+
+**Docker:**
+
+```bash
+# DockerHub (interactive prompt)
+docker login
+
+# NGC (use --password-stdin to avoid exposing secrets in process list)
+echo "$NGC_API_KEY" | docker login nvcr.io -u '$oauthtoken' --password-stdin
+```
+
+3. **Suggest an alternative image** on an authenticated registry. NVIDIA clusters typically have NGC auth pre-configured, so prefer NGC-hosted images:
+
+| DockerHub image | NGC alternative |
+| --- | --- |
+| `vllm/vllm-openai:latest` | `nvcr.io/nvidia/vllm:<YY.MM>-py3` (check [NGC catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm) for latest tag) |
+| `nvcr.io/nvidia/tensorrt-llm/release:<tag>` | Already NGC |
+
+> **Note:** NGC image tags follow `YY.MM-py3` format (e.g., `26.03-py3`). Not all DockerHub images have NGC equivalents. If no NGC alternative exists and DockerHub auth is missing, the user must add DockerHub credentials or pre-cache the image as a `.sqsh` file.
+
+4. After the user fixes auth or switches images, verify the image is **actually pullable** before submitting (credentials alone don't guarantee the image exists):
+
+```bash
+# enroot — test pull (aborts after manifest fetch)
+enroot import --output /dev/null docker://<registry>#<image> 2>&1 | head -10
+# Success: shows "Fetching image manifest" + layer info
+# Failure: shows "401 Unauthorized" or "404 Not Found"
+
+# docker
+docker manifest inspect <image> 2>&1 | head -5
+
+# singularity
+singularity pull --dry-run docker://<image> 2>&1 | head -5
+```
+
+> **Important**: Credentials existing for a registry does NOT mean a specific image is accessible. The image may not exist, or the credentials may lack permissions for that repository. Always verify the specific image before submitting.
+
+### Common failure modes
+
+| Symptom | Runtime | Cause | Fix |
+| --- | --- | --- | --- |
+| `curl: (22) ... error: 401` | enroot | No credentials for registry | Add to `~/.config/enroot/.credentials` |
+| `pyxis: failed to import docker image` | enroot | Auth failed or rate limit | Check credentials; DockerHub free: 100 pulls/6h per IP |
+| `unauthorized: authentication required` | docker | No `docker login` | Run `docker login [registry]` |
+| Image pulls on some nodes but not others | any | Cached on one node only | Pre-cache image or ensure auth on all nodes |

.claude/skills/debug/SKILL.md

@@ -0,0 +1,33 @@
+---
+name: debug
+description: Run commands inside a remote Docker container via the file-based command relay (tools/debugger). Use when the user says "run in Docker", "run on GPU", "debug remotely", "run test in container", "check nvidia-smi", "run pytest in Docker", or needs to execute any command inside a Docker container that shares the repo filesystem. Requires the user to have started server.sh inside the container first.
+---
+
+# Remote Docker Debugger
+
+Execute commands inside a Docker container from the host using the file-based command relay.
+
+**Read `tools/debugger/CLAUDE.md` for full usage details** — it has the protocol and examples.
+
+## Quick Reference
+
+```bash
+# Check connection
+bash tools/debugger/client.sh status
+
+# Connect to server (user must start server.sh in Docker first)
+bash tools/debugger/client.sh handshake
+
+# Run a command
+bash tools/debugger/client.sh run "<command>"
+
+# Long-running command (default timeout is 600s)
+bash tools/debugger/client.sh --timeout 1800 run "<command>"
+
+# Cancel the currently running command
+bash tools/debugger/client.sh cancel
+
+# Reconnect after server restart
+bash tools/debugger/client.sh flush
+bash tools/debugger/client.sh handshake
+```

.claude/skills/deployment/SKILL.md

@@ -174,6 +174,8 @@ All checks must pass before reporting success to the user.
 
 If a cluster config exists (`~/.config/modelopt/clusters.yaml` or `.claude/clusters.yaml`), or the user mentions running on a remote machine:
 
+0. **Check container registry auth** — before submitting any SLURM job with a container image, verify credentials exist on the cluster per `skills/common/slurm-setup.md` section 6. If credentials are missing for the image's registry, ask the user to fix auth or switch to an image on an authenticated registry (e.g., NGC). **Do not submit until auth is confirmed.**
+
 1. **Source remote utilities:**
 
    ```bash
@@ -222,6 +224,10 @@ For NEL-managed deployment (evaluation with self-deployment), use the evaluation
 | `Connection refused` on health check | Server still starting | Wait 30-60s for large models; check logs for errors |
 | `modelopt_fp4 not supported` | Framework doesn't support FP4 for this model | Check support matrix in `references/support-matrix.md` |
 
+## Unsupported Models
+
+If the model is not in the validated support matrix (`references/support-matrix.md`), deployment may fail due to weight key mismatches, missing architecture mappings, or quantized/unquantized layer confusion. Read `references/unsupported-models.md` for the iterative debug loop: **run → read error → diagnose → patch framework source → re-run**. For kernel-level issues, escalate to the framework team rather than attempting fixes.
+
 ## Success Criteria
 
 1. Server process is running and healthy (`/health` returns 200)

.claude/skills/deployment/references/unsupported-models.md

@@ -0,0 +1,70 @@
+# Deploying Unsupported Models
+
+When deploying a model not in the validated support matrix (`support-matrix.md`), expect failures. This guide covers the iterative debug loop for getting unsupported models running on vLLM, SGLang, or TRT-LLM.
+
+## Step 1 — Run and collect the error
+
+Submit the deployment job. When it fails, read the full log — focus on the **first** error traceback (not "See root cause above" wrappers). Identify the file and line number in the framework source.
+
+## Step 2 — Diagnose the root cause
+
+Fetch the framework source at the failing line (use `gh api` for the tagged version, or `find` inside the container). Common error categories:
+
+| Category | Symptoms | Examples |
+|----------|----------|----------|
+| **Weight key mismatch** | `KeyError`, `Unexpected key`, `Missing key` during weight loading | Checkpoint uses `model.language_model.layers.*` but framework expects `model.layers.*`. See [vllm#39406](https://github.com/vllm-project/vllm/pull/39406) |
+| **Quantized/unquantized layer confusion** | Wrong layer type loaded, dtype errors, shape mismatches | Framework tries to load unquantized layers with FP4 kernel due to overly broad `quantization_config.ignore` patterns or missing ignore entries. See [sglang#18937](https://github.com/sgl-project/sglang/pull/18937) |
+| **Missing architecture support** | `NoneType is not iterable`, `KeyError` on model type, unknown architecture | Framework's model handler doesn't recognize the text backbone type (e.g., `ministral3` not handled in vLLM's `mistral3.py` init). Fix: extend the model type mapping |
+| **Transformers version mismatch** | `ImportError`, `KeyError` on config fields | Framework ships with older transformers that doesn't know the model type. Fix: upgrade transformers after installing the framework |
+| **Kernel-level issues** | CUDA errors, `triton` import failures, unsupported ops | Framework lacks kernel support for this model + quantization combo |
+
+## Step 3 — Apply a targeted fix
+
+Focus on **small, targeted patches** to the framework source. Do not modify `config.json` or the checkpoint — fix the framework's handling instead.
+
+### Weight key mismatches and architecture mapping gaps
+
+Patch the framework source in the run script using `sed` or a Python one-liner. Keep patches minimal — change only what's needed to unblock the current error.
+
+```bash
+# Example: extend model type mapping in vLLM mistral3.py
+FRAMEWORK_FILE=$(find /usr/local/lib -path "*/vllm/model_executor/models/mistral3.py" 2>/dev/null | head -1)
+sed -i 's/old_pattern/new_pattern/' "${FRAMEWORK_FILE}"
+```
+
+> **Tip**: when locating framework source files inside containers, use `find` instead of Python import — some frameworks print log messages to stdout during import that can corrupt captured paths.
+
+### Speeding up debug iterations (vLLM)
+
+When iterating on fixes, use these flags to shorten the feedback loop:
+
+- **`--load-format dummy`** — skip loading actual model weights. Useful for testing whether the model initializes, config is parsed correctly, and weight keys match without waiting for the full checkpoint load.
+- **`VLLM_USE_PRECOMPILED=1 pip install --editable .`** — when patching vLLM source directly (instead of `sed`), this rebuilds only Python code without recompiling C++/CUDA extensions.
+
+### Quantized/unquantized layer confusion
+
+Check `hf_quant_config.json` ignore patterns against the framework's weight loading logic. The framework may try to load layers listed in `ignore` with quantized kernels, or vice versa. Fix by adjusting the framework's layer filtering logic.
+
+### Kernel-level issues
+
+These require framework kernel team involvement. Do NOT attempt to patch kernels. Instead:
+
+1. Document the exact error (model, format, framework version, GPU type)
+2. Inform the user: *"This model + quantization combination requires kernel support that isn't available in {framework} v{version}. I'd suggest reaching out to the {framework} kernel team or trying a different framework."*
+3. Suggest trying an alternative framework (vLLM → SGLang → TRT-LLM)
+
+## Step 4 — Re-run and iterate
+
+After applying a fix, resubmit the job. Each iteration may reveal a new error (e.g., fixing the init error exposes a weight loading error). Continue the loop: **run → read error → diagnose → patch → re-run**.
+
+Typical iteration count: 1-3 for straightforward fixes, 3-5 for models requiring multiple patches.
+
+## Step 5 — Know when to stop
+
+**Stop patching and escalate** when:
+
+- The error is in compiled CUDA kernels or triton ops (not Python-level)
+- The fix requires changes to core framework abstractions (not just model handlers)
+- You've done 5+ iterations without the server starting
+
+In these cases, inform the user and suggest: trying a different framework, checking for a newer framework version, or filing an issue with the framework team.

.claude/skills/evaluation/SKILL.md

@@ -28,6 +28,7 @@ Config Generation Progress:
 - [ ] Step 5: Confirm tasks (iterative)
 - [ ] Step 6: Advanced - Multi-node (Data Parallel)
 - [ ] Step 7: Advanced - Interceptors
+- [ ] Step 7.5: Check container registry auth (SLURM only)
 - [ ] Step 8: Run the evaluation
 ```
 
@@ -74,9 +75,9 @@ Prompt the user with "I'll ask you 5 questions to build the base config we'll ad
 4. Safety & Security (like Garak and Safety Harness)
 5. Multilingual (like MMATH, Global MMLU, MMLU-Prox)
 
-DON'T ALLOW FOR ANY OTHER OPTIONS, only the ones listed above under each category (Execution, Deployment, Auto-export, Model type, Benchmarks). YOU HAVE TO GATHER THE ANSWERS for the 5 questions before you can build the base config.
+Only accept options from the categories listed above (Execution, Deployment, Auto-export, Model type, Benchmarks). YOU HAVE TO GATHER THE ANSWERS for the 5 questions before you can build the base config.
 
-> **Note:** These categories come from NEL's `build-config` CLI. If `nel skills build-config --help` shows different options than listed above, use the CLI's current options instead.
+> **Note:** These categories come from NEL's `build-config` CLI. **Always run `nel skills build-config --help` first** to get the current options — they may differ from this list (e.g., `chat_reasoning` instead of separate `chat`/`reasoning`, `general_knowledge` instead of `standard`). When the CLI's current options differ from this list, prefer the CLI's options.
 
 When you have all the answers, run the script to build the base config:
 
@@ -181,6 +182,36 @@ If the user needs multi-node evaluation (model >120B, or more throughput), read
 
 - The docs may show incorrect parameter names for logging. Use `max_logged_requests` and `max_logged_responses` (NOT `max_saved_*` or `max_*`).
 
+**Step 7.5: Check container registry authentication (SLURM only)**
+
+NEL's default deployment images by framework:
+
+| Framework | Default image | Registry |
+| --- | --- | --- |
+| vLLM | `vllm/vllm-openai:latest` | DockerHub |
+| SGLang | `lmsysorg/sglang:latest` | DockerHub |
+| TRT-LLM | `nvcr.io/nvidia/tensorrt-llm/release:...` | NGC |
+| Evaluation tasks | `nvcr.io/nvidia/eval-factory/*:26.03` | NGC |
+
+Before submitting, verify the cluster has credentials for the deployment image. See `skills/common/slurm-setup.md` section 6 for the full procedure.
+
+```bash
+ssh <host> "grep -E '^\s*machine\s+' ~/.config/enroot/.credentials 2>/dev/null"
+```
+
+**Decision flow (check before submitting):**
+1. Check if the cluster has credentials for the default DockerHub image (see command above)
+2. If DockerHub credentials exist → use the default image and submit
+3. If DockerHub credentials are missing but can be added → add them (see `slurm-setup.md` section 6), then submit
+4. If DockerHub credentials cannot be added → override `deployment.image` to the NGC alternative and submit:
+
+   ```yaml
+   deployment:
+     image: nvcr.io/nvidia/vllm:<YY.MM>-py3  # check https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm for latest tag
+   ```
+
+5. **Do not retry more than once** without fixing the auth issue
+
 **Step 8: Run the evaluation**
 
 Print the following commands to the user. Propose to execute them in order to confirm the config works as expected before the full run.
@@ -303,5 +334,6 @@ Config Generation Progress:
 - [ ] Step 5: Confirm tasks (iterative)
 - [ ] Step 6: Advanced - Multi-node (Data Parallel)
 - [ ] Step 7: Advanced - Interceptors
+- [ ] Step 7.5: Check container registry auth (SLURM only)
 - [ ] Step 8: Run the evaluation
 ```

.claude/skills/ptq/SKILL.md

@@ -24,6 +24,24 @@ Check the support table in `examples/llm_ptq/README.md` for verified HF models.
 - **Listed** → supported, use `hf_ptq.py` (step 4A/4B)
 - **Not listed** → read `references/unsupported-models.md` to determine if `hf_ptq.py` can still work or if a custom script is needed (step 4C)
 
+## Step 2.5 — Check for model-specific dependencies
+
+If the model uses `trust_remote_code` (check `config.json` for `auto_map`), inspect its custom Python files for imports not present in the container:
+
+```bash
+grep -h "^from \|^import " <model_path>/modeling_*.py | sort -u
+```
+
+**Known dependency patterns:**
+
+| Import found | Packages to install |
+| --- | --- |
+| `from mamba_ssm` / `from causal_conv1d` | `mamba-ssm causal-conv1d` (Mamba/hybrid models: NemotronH, Jamba) |
+
+If extra deps are needed:
+- **Launcher (4B)**: set `EXTRA_PIP_DEPS` in the task's `environment` section — `ptq.sh` installs them automatically
+- **Manual (4A)**: `unset PIP_CONSTRAINT && pip install <deps>` before running `hf_ptq.py`
+
 ## Step 3 — Choose quantization format
 
 **First**, check for a model-specific recipe:
@@ -113,6 +131,10 @@ ls -lh <output_path>/
 
 Report the path and size to the user.
 
+### Post-quantization validation
+
+Validate the exported checkpoint's quantization pattern matches the recipe. Quantization config patterns can silently miss layers if the model uses non-standard naming (e.g., Gemma4 `experts.*` missed by `*mlp*` patterns) — this only surfaces later as deployment failures. Read `references/checkpoint-validation.md` for the validation script, expected patterns per recipe, and common pattern gaps.
+
 ## Key API Rules
 
 - `mtq.register()` classes **must** define `_setup()` and call it from `__init__`
@@ -124,6 +146,7 @@ Report the path and size to the user.
 
 ## Common Pitfalls
 
+- **Model-specific dependencies**: Models with `trust_remote_code` may import packages not in the container (e.g., `mamba-ssm` for hybrid Mamba models). See Step 2.5. Use `EXTRA_PIP_DEPS` env var with the launcher, or install manually before running `hf_ptq.py`
 - **Transformers version**: New models may need a newer version of transformers than what's installed. Check `config.json` for `transformers_version`. In containers, beware of `PIP_CONSTRAINT` blocking upgrades — see `references/slurm-setup-ptq.md` for workarounds
 - **Gated datasets**: Some calibration datasets require HF authentication. Ensure `HF_TOKEN` is set in the job environment, or use `--dataset cnn_dailymail` as a non-gated alternative
 - **NFS root_squash + Docker**: See `skills/common/slurm-setup.md` section 5
@@ -137,6 +160,7 @@ Report the path and size to the user.
 | `references/launcher-guide.md` | Step 4B only (launcher path) |
 | `tools/launcher/CLAUDE.md` | Step 4B only, if you need more launcher detail |
 | `references/unsupported-models.md` | Step 4C only (unlisted model) |
+| `references/checkpoint-validation.md` | Step 5: validate quantization pattern matches recipe |
 | `skills/common/remote-execution.md` | Step 4A/4C only, if target is remote |
 | `skills/common/slurm-setup.md` | Step 4A/4C only, if using SLURM manually (not launcher) |
 | `references/slurm-setup-ptq.md` | Step 4A/4C only, PTQ-specific SLURM (container, GPU sizing, FSDP2) |