Add a standalone monitor skill for persistent job tracking (#1252)

### What does this PR do?

Type of change: ? <!-- Use one of the following: Bug fix, new feature,
new example, new tests, documentation. -->
Add a standalone monitor skill for persistent job tracking across
sessions, and integrate it with PTQ, evaluation, and deployment skills.

Problem: Each skill had ad-hoc inline monitoring (squeue polling, nel
status checks) that didn't survive session restarts and couldn't track
multiple jobs. Users had to manually ask "check status" every time.

Solution: A centralized monitor skill with:
- Job registry (.claude/active_jobs.json): single source of truth for
all active jobs
- Durable recurring cron: polls every 15 min, survives session restarts,
self-cleans when all jobs complete
- User-initiated mode: works in new conversations by reading the
registry
- Aggregated reporting: "2 of 4 completed" instead of per-job noise

### Usage
After any skill submits a job, the monitor skill automatically:

1. Registers the job in .claude/active_jobs.json
2. Sets up a durable cron to poll status every 15 minutes

User can also trigger manually:
User: "check my eval status" → reads registry, reports current state
User: "is the PTQ done?"            → finds job, checks status
User: "what jobs are running?"      → lists all registered jobs

### Testing
<!-- Mention how have you tested your change if applicable. -->

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅ / ❌ / N/A <!--- If ❌, explain
why. -->
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ / ❌ / N/A
<!--- Mandatory -->
- Did you write any new necessary tests?: ✅ / ❌ / N/A <!--- Mandatory
for new features or examples. -->
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅ / ❌ / N/A <!--- Only for new features, API changes, critical bug fixes
or backward incompatible changes. -->

### Additional Information
<!-- E.g. related issue. -->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added monitor skill for tracking SLURM jobs, NEL evaluations, and
launcher experiments with persistent job registry.

* **Documentation**
* Updated deployment, evaluation, and PTQ documentation to use the new
monitor skill.
  * Simplified diagnostic and troubleshooting instructions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Kai Xu <kaix@nvidia.com>

Author: kaix-nv

.claude/skills/deployment/SKILL.md

@@ -195,7 +195,7 @@ If a cluster config exists (`~/.config/modelopt/clusters.yaml` or `.claude/clust
 
 3. **Deploy based on remote environment:**
 
-   - **SLURM** — see `skills/common/slurm-setup.md` for job script templates (container setup, account/partition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model <path> --quantization modelopt`). Use `remote_submit_job` and `remote_poll_job` to manage the job. Get the node hostname from `squeue -j $JOBID -o %N`.
+   - **SLURM** — see `skills/common/slurm-setup.md` for job script templates (container setup, account/partition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model <path> --quantization modelopt`). After submitting, register the job and set up monitoring per the **monitor skill**. Get the node hostname from `squeue -j $JOBID -o %N`.
 
    - **Bare metal / Docker** — use `remote_run` to start the server directly:
 

.claude/skills/evaluation/SKILL.md

@@ -256,64 +256,24 @@ After the dry-run, check the output from `nel` for any problems with the config.
 
 **Monitoring Progress**
 
-After job submission, you can monitor progress using:
+After job submission, register the job per the **monitor skill** for durable cross-session tracking. For one-off queries (live status, debugging a failed run, analyzing results) use the **launching-evals skill**; for querying past runs in MLflow use **accessing-mlflow**.
 
-1. **Check job status:**
+**NEL-specific diagnostics** (for debugging failures):
 
-   ```bash
-   nel status <invocation_id>
-   nel info <invocation_id>
-   ```
-
-2. **Stream logs** (Local execution only):
-
-   ```bash
-   nel logs <invocation_id>
-   ```
-
-   Note: `nel logs` is not supported for SLURM execution.
-
-3. **Inspect logs via SSH** (SLURM workaround):
-
-   When `nel logs` is unavailable (SLURM), use SSH to inspect logs directly:
-
-   First, get log locations:
-
-   ```bash
-   nel info <invocation_id> --logs
-   ```
-
-   Then, use SSH to view logs:
-
-   **Check server deployment logs:**
-
-   ```bash
-   ssh <username>@<hostname> "tail -100 <log path from `nel info <invocation_id> --logs`>/server-<slurm_job_id>-*.log"
-   ```
-
-   Shows vLLM server startup, model loading, and deployment errors (e.g., missing wget/curl).
-
-   **Check evaluation client logs:**
-
-   ```bash
-   ssh <username>@<hostname> "tail -100 <log path from `nel info <invocation_id> --logs`>/client-<slurm_job_id>.log"
-   ```
-
-   Shows evaluation progress, task execution, and results.
-
-   **Check SLURM scheduler logs:**
-
-   ```bash
-   ssh <username>@<hostname> "tail -100 <log path from `nel info <invocation_id> --logs`>/slurm-<slurm_job_id>.log"
-   ```
-
-   Shows job scheduling, health checks, and overall execution flow.
-
-   **Search for errors:**
-
-   ```bash
-   ssh <username>@<hostname> "grep -i 'error\|warning\|failed' <log path from `nel info <invocation_id> --logs`>/*.log"
-   ```
+```bash
+# Quick status check
+nel status <invocation_id>
+nel info <invocation_id>
+
+# Get log paths
+nel info <invocation_id> --logs
+
+# Inspect logs via SSH
+ssh <user>@<host> "tail -100 <log_path>/server-<slurm_job_id>-*.log"   # deployment errors
+ssh <user>@<host> "tail -100 <log_path>/client-<slurm_job_id>.log"     # evaluation errors
+ssh <user>@<host> "tail -100 <log_path>/slurm-<slurm_job_id>.log"      # scheduling/walltime
+ssh <user>@<host> "grep -i 'error\|failed' <log_path>/*.log"           # search all logs
+```
 
 ---
 

.claude/skills/monitor/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: monitor
+description: Monitor submitted jobs (PTQ, evaluation, deployment) on SLURM clusters. Use when the user asks "check job status", "is my job done", "monitor my evaluation", "what's the status of the PTQ", "check on job <slurm_job_id>", or after any skill submits a long-running job. Also triggers on "nel status", "squeue", or any request to check progress of a previously submitted job.
+---
+
+# Job Monitor
+
+Monitor jobs submitted to SLURM clusters — PTQ quantization, NEL evaluation, model deployment, or raw SLURM jobs.
+
+## When to use
+
+1. **Auto-monitor** — another skill (PTQ, evaluation, deployment) just submitted a job. Register the job and set up monitoring immediately.
+2. **User-initiated** — user asks about a job status, possibly in a new conversation. Check the registry, identify the job, and report.
+
+---
+
+## Job Registry
+
+All active jobs are tracked in `.claude/active_jobs.json`. This file is the single source of truth for what's being monitored.
+
+```json
+[
+  {
+    "type": "nel",
+    "id": "<invocation_id or slurm_job_id>",
+    "host": "<cluster_hostname>",
+    "user": "<ssh_user>",
+    "submitted": "YYYY-MM-DD HH:MM",
+    "description": "<what this job does>",
+    "last_status": "<last known status>"
+  }
+]
+```
+
+`type` is one of: `nel`, `slurm`, `launcher`.
+
+---
+
+## On Job Submission
+
+Every time a job is submitted (by any skill or manually):
+
+1. **Add an entry** to `.claude/active_jobs.json`. Create the file if it doesn't exist.
+2. **Set up a durable recurring cron** (if one isn't already running) that polls all registered jobs every 15 minutes. The cron prompt should: read the registry, check each job, report state changes to the user, remove completed jobs, and delete itself when the registry is empty.
+
+Always do both steps. Don't try to predict job duration.
+
+---
+
+## On Cron Fire / Status Check
+
+Whether triggered by the cron or by the user asking "check status":
+
+1. **Read the registry** from `.claude/active_jobs.json`
+2. **Check each job** using the appropriate method (see below)
+3. **Report only state changes** — compare against `last_status` in registry
+4. **Update `last_status`** in the registry
+5. **Remove completed jobs** — any job in a terminal state (COMPLETED, FAILED, CANCELLED, KILLED)
+6. **If registry is empty** — delete the recurring cron
+
+---
+
+## How to Check Each Job Type
+
+### NEL jobs (`type: nel`)
+
+- **Check:** `nel status <id>`
+- **On completion:** `nel info <id>` to fetch results
+- **On failure:** `nel info <id> --logs` then inspect server/client/SLURM logs via SSH
+
+### Launcher jobs (`type: launcher`)
+
+- **Check:** Tail the launcher's background output file for key events
+- **Key events:** experiment ID, SLURM job ID, container import, calibration progress, export path, final status
+- **On failure:** Look for `Traceback`, `Error`, or `FAILED` in the output
+
+### Raw SLURM jobs (`type: slurm`)
+
+- **Check:** `ssh <host> "squeue -j <id> -h -o '%T %M %R'"` — if empty, job left the queue
+- **On completion:** `ssh <host> "sacct -j <id> --format=State,ExitCode,Elapsed -n"`
+- **On failure:** Check the job's output log file
+
+---
+
+## Identifying Jobs (user-initiated, no ID given)
+
+When the user asks about a job without specifying an ID, check in order:
+
+1. `.claude/active_jobs.json` — most reliable, has context
+2. `nel ls runs --since 1d` — recent NEL runs
+3. `ssh <host> "squeue -u <user>"` — active SLURM jobs
+4. `ls -lt tools/launcher/experiments/cicd/ | head -10` — recent launcher experiments
+
+---
+
+## Reporting Guidelines
+
+- **Report state changes proactively** — PENDING → RUNNING, or job completes
+- **Aggregate multiple jobs** — "2 of 4 completed (MMLU-Pro: 42.3%, GSM8K: 67.1%), 1 running, 1 pending"
+- **Summarize, don't echo** — interpret events ("Calibration complete, exporting checkpoint") not raw logs
+- **On failure, diagnose immediately** — check logs and report root cause without waiting for user to ask
+- **Minimize noise** — don't report "still running" unless the user is actively asking

.claude/skills/ptq/SKILL.md

@@ -118,9 +118,7 @@ For SLURM, see `skills/common/slurm-setup.md` and `references/slurm-setup-ptq.md
 
 ### Monitoring
 
-- **Launcher**: blocks and tails logs automatically
-- **SLURM (manual)**: poll with `squeue -u $USER` + `sleep` (not cron or background tasks)
-- **Local**: watch stdout
+After job submission, register the job and set up monitoring per the **monitor skill**.
 
 ## Step 5 — Verify output