Add monitor_training + calibrate_grader scripts, fix 4 bugs (#30)

* Add monitor + calibrate scripts, fix 4 bugs from code review

New scripts:
- monitor_training.py: Poll a running job until completion, stream events
  in real time with status icons for reward steps, errors, and milestones
- calibrate_grader.py: Run base model on training data, score with your
  Python grader, output pass rates at every threshold, recommend optimal
  pass_threshold. Automates the most critical pre-RFT step.

Bug fixes (from code review, missed in PR #29 squash):
- submit_training.py, deploy_model.py: Add missing 'requests' to PEP 723 deps
- submit_training.py: Fix file handle leak on grader file read
- evaluate_model.py: Fix StopIteration crash on malformed test data
- validate/validate_rft.py: Fix broken newline escape detection logic

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add quickstart guide: 5-step path to first fine-tuned model

New workflow: quickstart.md — goes from zero to a deployed fine-tuned
model in 5 steps (credentials, data prep, submit, monitor, deploy+test).
Targets novice users who just want to get started without reading the
full pipeline guide.

SKILL.md updated to list quickstart as the first workflow option.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Revise quickstart: synthetic data from prompt, baseline step, comparison

- Step 2: Generate training data from a prompt (no pre-existing data needed)
- Step 3: Baseline the base model before training (see what you're improving)
- Step 6: Side-by-side comparison of base vs fine-tuned on same questions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add cleanup.py: list and delete old files, deployments, pending jobs

Helps users avoid quota exhaustion (max 100 files per resource). Supports:
- --list (all/jobs/deployments/files) for resource inventory
- --delete-files with optional --older-than N days filter
- --cancel-pending to cancel queued jobs
- --dry-run to preview before executing

Includes quota warning when approaching 80/100 file limit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix: JSON parsing for LLM output, align failure rate docs to 25-50%

1. quickstart.md: Use regex to extract JSON from code fences — the
   simple strip() approach fails when LLM adds preamble text before
   the JSON block
2. grader-design.md: Change prose from '30-50%' to '25-50%' to match
   the table and calibrate_grader.py code

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update README, CHANGELOG, and GETTING_STARTED for agent skills

README.md:
- Add quickstart + agent-assisted paths to Quick Start section
- Update AI Agent Skills counts (12 scripts, 14 references, 6 workflows)

CHANGELOG.md:
- Add entries for PRs #24, #26, #28, #29, #30
- Restructure into versioned releases (1.0.0, 1.1.0, 2.0.0, Unreleased)

GETTING_STARTED.md:
- Add Option A (AI-Assisted) path before the manual demo path
- Fix repo URL (Azure/fine-tuning -> microsoft-foundry/fine-tuning)
- Renumber steps to accommodate new section

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update Skills/README.md: auto-discovery, new scripts, uv support, grader guidance

- Quick Start: add auto-discovery path (symlinks), uv as recommended install
- What This Skill Covers: add grader calibration, monitoring, cleanup, token cost
- Directory Structure: add monitor_training, calibrate_grader, cleanup, quickstart,
  grader-design; update descriptions for accuracy
- Guidance Highlights: add grader calibration, Python grader default, token cost

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* CHANGELOG: set Unreleased to v2.1.0 (2026-04-21)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix: SKILL.md missing DPO in submit description, remove unused requests dep from cleanup.py

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: AliciaFrame

CHANGELOG.md

@@ -4,7 +4,48 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## [Unreleased]
+## [2.1.0] - 2026-04-21
+
+### Added
+- `monitor_training.py` — poll a running job until completion with real-time event streaming
+- `calibrate_grader.py` — calibrate RFT grader pass_threshold on base model outputs
+- `cleanup.py` — list and delete old files, deployments, and pending jobs to reclaim quota
+- `workflows/quickstart.md` — 6-step guide from zero to a deployed fine-tuned model
+- `references/grader-design.md` — comprehensive RFT grader design guide (type selection, partial credit, threshold calibration)
+- RFT-specific metrics section in `references/training-curve-analysis.md` (reward curves, token growth, parse errors, checkpoint selection)
+- LLM augmentation approach in `workflows/dataset-creation.md`
+- Agent discovery symlinks (`.github/skills/`, `.claude/skills/`, `.agents/skills/`) for auto-discovery by Copilot, Claude, and Codex
+- PEP 723 inline script metadata on all scripts for `uv run` support
+- `HelpOnErrorParser` for better CLI error messages across all scripts
+- `DefaultAzureCredential` fallback in `common.py` for keyless authentication
+- AI Agent Skills section in README with usage instructions for all supported agents
+
+### Changed
+- Updated hyperparameter recommendations in `references/agentic-rft.md` (LR=1.0, compute_multiplier=1.5)
+- Restructured `workflows/dataset-creation.md` into three equal approaches (manual, LLM augmentation, synthetic generation)
+- Updated `references/evaluation-methodology.md` to include token cost alongside accuracy
+- Generalized experiment-specific numbers across all docs
+
+### Fixed
+- `submit_training.py`, `deploy_model.py`: Added missing `requests` to PEP 723 dependencies
+- `submit_training.py`: Fixed file handle leak when reading grader files
+- `evaluate_model.py`: Fixed `StopIteration` crash on malformed test data
+- `validate/validate_rft.py`: Fixed broken newline escape detection logic
+- `convert_dataset.py`: Fixed DPO generation to include system messages for base model
+
+## [2.0.0] - 2026-04-16
+
+### Added
+- **Azure AI Fine-Tuning coding agent skill** (`Skills/`) — comprehensive agent skill for SFT, DPO, and RFT workflows on Azure AI Foundry
+  - `SKILL.md` with full lifecycle guidance, platform gotchas, and rules
+  - 8 reusable Python scripts (submit, check, deploy, evaluate, convert, generate, score, common)
+  - Data validators for SFT, DPO, and RFT formats with cost estimates
+  - 11 reference docs covering hyperparameters, dataset formats, deployment, evaluation, agentic RFT, reward hacking prevention, vision FT, cost management, and more
+  - 5 guided workflows (full pipeline, dataset creation, iterative training, diagnosis, experiment review)
+  - Sample data for SFT, DPO, and RFT formats
+- RFT best practice guidance (PR #24, contributed by Blanca Li)
+
+## [1.1.0] - 2026-02-02
 
 ### Added
 - Enhanced main README with navigation table, quickstart section, and demo links
@@ -22,8 +63,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Fixed
 - Fixed placeholder URLs in CONTRIBUTING.md
 
----
-
 ## [1.0.0] - 2025-01-15
 
 ### Added

GETTING_STARTED.md

@@ -33,13 +33,33 @@ Ensure you have these roles on your Azure AI Foundry project:
 ## Step 1: Clone the Repository
 
 ```bash
-git clone https://github.com/Azure/fine-tuning.git
+git clone https://github.com/microsoft-foundry/fine-tuning.git
 cd fine-tuning
 ```
 
 ---
 
-## Step 2: Choose Your First Demo
+## Step 2: Choose Your Path
+
+### Option A: AI-Assisted (Recommended)
+
+If you have **GitHub Copilot**, **Claude Code**, or another coding agent:
+
+1. Open this repo in your editor (VS Code, cursor, etc.)
+2. Ask your coding agent: *"Help me fine-tune a model on my data"*
+3. The agent auto-discovers the fine-tuning skill and guides you through:
+   - Data preparation and validation
+   - Job submission with correct parameters
+   - Monitoring and training curve analysis
+   - Deployment and evaluation
+
+Or follow the step-by-step quickstart without an agent: **[Skills/workflows/quickstart.md](Skills/workflows/quickstart.md)**
+
+### Option B: Follow a Demo Notebook
+
+If you prefer to follow a notebook manually, continue below.
+
+## Step 3: Choose Your First Demo
 
 We recommend starting with **Supervised Fine-Tuning (SFT)** as it covers the most common use cases.
 
@@ -53,7 +73,7 @@ We recommend starting with **Supervised Fine-Tuning (SFT)** as it covers the mos
 
 ---
 
-## Step 3: Set Up the Demo Environment
+## Step 4: Set Up the Demo Environment
 
 Navigate to your chosen demo folder:
 
@@ -93,7 +113,7 @@ pip install -r requirements.txt
 
 ---
 
-## Step 4: Run the Notebook
+## Step 5: Run the Notebook
 
 1. Open the notebook in VS Code or Jupyter:
    ```bash
@@ -110,7 +130,7 @@ pip install -r requirements.txt
 
 ---
 
-## Step 5: Understand the Data Format
+## Step 6: Understand the Data Format
 
 Each fine-tuning technique requires a specific data format. See the schema documentation:
 

README.md

@@ -20,6 +20,10 @@ This repository contains **12 end-to-end demos** and **sample datasets** for fin
 1. **[SFT_CNN_DailyMail](Demos/SFT_CNN_DailyMail/)** - Best first demo! Learn Supervised Fine-Tuning with news summarization
 2. **[Sample_Datasets](Sample_Datasets/)** - Understand data formats for SFT, DPO, and RFT
 
+**Want an AI coding assistant to guide you?**
+- Open this repo in VS Code with Copilot, or use Claude/Codex — the agent skill auto-discovers and walks you through the full workflow
+- Or follow **[Skills/workflows/quickstart.md](Skills/workflows/quickstart.md)** — fine-tune your first model in 6 steps (no demo notebook needed)
+
 **Ready for advanced techniques?**
 - **[DPO_Intel_Orca](Demos/DPO_Intel_Orca/)** - Direct Preference Optimization
 - **[RFT_Countdown](Demos/RFT_Countdown/)** - Reinforcement Fine-Tuning
@@ -79,9 +83,9 @@ This repo includes a fine-tuning skill that coding agents can auto-discover and
 
 All three paths are symlinks to the canonical skill at **[Skills/](Skills/)**, which includes:
 - **SKILL.md** — Agent instructions covering SFT, DPO, and RFT workflows
-- **8 scripts** — submit, monitor, deploy, evaluate, validate, score, convert, distill
-- **11 reference docs** — hyperparameters, dataset formats, agentic RFT, cost management, and more
-- **5 guided workflows** — full pipeline, dataset creation, iterative training, diagnosis
+- **12 scripts** — submit, monitor, calibrate, check, deploy, evaluate, validate, score, convert, generate, cleanup, and shared utilities
+- **14 reference docs** — grader design, hyperparameters, dataset formats, agentic RFT, cost management, and more
+- **6 guided workflows** — quickstart, full pipeline, dataset creation, iterative training, diagnosis, experiment review
 - **Sample data** — SFT, DPO, and RFT example JSONL files
 
 ### Using with GitHub Copilot (VS Code)

Skills/README.md

@@ -10,45 +10,48 @@ A **skill** is a structured set of instructions, reference documentation, and re
 
 | Stage | What the agent does |
 |-------|-------------------|
-| **Dataset creation** | Generate synthetic training data using Data Designer, or prepare existing data |
+| **Dataset creation** | Generate synthetic training data, curate examples, or augment with LLM rephrasings |
 | **Dataset validation** | Validate JSONL schema, token limits, and format for SFT / DPO / RFT |
 | **Base model evaluation** | Benchmark the un-tuned model to establish a baseline |
 | **Training type selection** | Choose between SFT, DPO, and RFT based on your task |
+| **Grader calibration** | For RFT: test your grader on base model outputs and find the optimal pass_threshold |
 | **Job submission** | Submit training jobs via SDK, REST API, or `azd` CLI |
-| **Training curve analysis** | Detect overfitting, recommend checkpoints |
+| **Job monitoring** | Poll running jobs with real-time event streaming |
+| **Training curve analysis** | Detect overfitting, monitor token growth, recommend checkpoints |
 | **Iterative experimentation** | Plan successive runs based on results |
 | **Model deployment** | Deploy fine-tuned models with the correct format and SKU |
-| **Model evaluation** | Score outputs with custom LLM judges |
+| **Model evaluation** | Score outputs with custom LLM judges, compare accuracy and token cost |
+| **Resource cleanup** | Delete old files and deployments to reclaim quota |
 
 ## Quick Start
 
-### 1. Install the skill
+### 1. Set up the skill
 
-**GitHub Copilot (VS Code / CLI):**
-Copy the `Skills/` directory into your project and reference `SKILL.md` in your Copilot instructions, or use it as a [Copilot custom instructions](https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions).
+**Auto-discovery (recommended):**
+If you cloned the [microsoft-foundry/fine-tuning](https://github.com/microsoft-foundry/fine-tuning) repo, coding agents auto-discover the skill via symlinks:
+- GitHub Copilot → `.github/skills/azure-ai-fine-tuning`
+- Claude Code → `.claude/skills/azure-ai-fine-tuning`
+- Codex / other agents → `.agents/skills/azure-ai-fine-tuning`
 
-**Claude Code:**
-Add the skill directory to your project and reference `SKILL.md` in your `CLAUDE.md` or system prompt.
+Just open the repo and start asking questions — no manual setup needed.
 
-**Any agent:**
-Point your agent at `SKILL.md` as a context file. It will discover the workflows, references, and scripts from there.
+**Manual setup:**
+Copy the `Skills/` directory into your project and reference `SKILL.md` in your agent's instructions file (`copilot-instructions.md`, `CLAUDE.md`, etc.).
 
 ### 2. Set up your environment
 
+**Option A: `uv` (zero-setup, recommended):**
+All scripts have [PEP 723](https://peps.python.org/pep-0723/) inline dependency declarations. Just run them with `uv`:
 ```bash
-cp Skills/.env.template Skills/.env
-# Edit .env with your Azure OpenAI endpoint, API key, and resource coordinates
+uv run Skills/scripts/submit_training.py --help
 ```
 
-Required Python packages:
+**Option B: Manual install:**
 ```bash
-pip install openai azure-identity tiktoken requests
-```
+cp Skills/.env.template Skills/.env
+# Edit .env with your Azure AI Foundry endpoint and API key
 
-Optional (for specific workflows):
-```bash
-pip install pandas pyarrow    # Dataset conversion and statistics
-pip install azure-ai-evaluation azure-ai-projects  # Azure AI Evaluation SDK integration
+pip install openai azure-identity requests
 ```
 
 ### 3. Start fine-tuning
@@ -64,7 +67,7 @@ The agent will read the appropriate workflow and guide you step by step.
 
 ```
 Skills/
-├── SKILL.md                          # Main skill file (entry point)
+├── SKILL.md                          # Main skill file (agent entry point)
 ├── .env.template                     # Environment variable template
 ├── README.md                         # This file
 ├── references/
@@ -73,7 +76,8 @@ Skills/
 │   ├── dataset-formats.md            # JSONL format specs for each training type
 │   ├── deployment-formats.md         # Model format, SKU, and version mapping
 │   ├── evaluation-methodology.md     # Eval rubric design and grader types
-│   ├── training-curve-analysis.md    # Reading training logs and curves
+│   ├── training-curve-analysis.md    # Reading training logs and curves (SFT + RFT)
+│   ├── grader-design.md             # RFT grader design (type selection, partial credit, calibration)
 │   ├── foundry-cli.md               # azd ai finetuning CLI reference
 │   ├── vision-fine-tuning.md         # Image/video fine-tuning (gpt-4o, gpt-4.1)
 │   ├── cost-management.md            # Training costs and budget planning
@@ -82,20 +86,24 @@ Skills/
 │   ├── reward-hacking-prevention.md  # Preventing reward hacking in RFT
 │   └── platform-bugs.md             # Known platform bugs and workarounds
 ├── workflows/
-│   ├── full-pipeline.md              # End-to-end workflow (start here)
-│   ├── dataset-creation.md           # Data generation with Data Designer
+│   ├── quickstart.md                 # 6-step quickstart (fine-tune your first model)
+│   ├── full-pipeline.md              # End-to-end workflow
+│   ├── dataset-creation.md           # Data generation (manual, LLM augmentation, synthetic)
 │   ├── iterative-training.md         # Training and HP tuning loop
 │   ├── diagnose-poor-results.md      # Troubleshooting bad results
 │   └── experiment-review.md          # Post-experiment review and next steps
 ├── scripts/
-│   ├── submit_training.py            # Submit SFT/RFT jobs (SDK + REST fallback)
-│   ├── generate_distillation_data.py # Generate distillation training data
-│   ├── check_training.py             # Training curve analysis
+│   ├── submit_training.py            # Submit SFT/DPO/RFT jobs (SDK + REST fallback)
+│   ├── monitor_training.py           # Poll a running job until completion
+│   ├── calibrate_grader.py           # RFT grader threshold calibration
+│   ├── check_training.py             # Training curve analysis and checkpoints
 │   ├── deploy_model.py               # Deploy via ARM REST API
 │   ├── evaluate_model.py             # LLM judge evaluation
 │   ├── convert_dataset.py            # Format conversion (SFT↔DPO↔RFT)
+│   ├── generate_distillation_data.py # Generate distillation training data
 │   ├── score_dataset.py              # Dataset quality scoring
-│   ├── common.py                     # Shared auth helpers
+│   ├── cleanup.py                    # Delete old files, deployments, pending jobs
+│   ├── common.py                     # Shared auth helpers + HelpOnErrorParser
 │   └── validate/
 │       ├── validate_sft.py           # SFT JSONL validator
 │       ├── validate_dpo.py           # DPO JSONL validator
@@ -137,8 +145,10 @@ Key patterns encoded in this skill:
 
 - **Start with SFT distillation** — the most reliable fine-tuning pattern, achieving high teacher gap closure with 200–500 examples
 - **Always baseline first** — evaluate the base model before fine-tuning to confirm there's room for improvement
+- **Calibrate your RFT grader** — target 25-50% failure rate on the base model; recalibrate when you change your dataset
+- **Python grader as default for RFT** — fast, deterministic, and reliable; endpoint graders only when you need external API calls during grading
 - **DPO is risky when the base is already strong** — it can degrade quality if the model already handles the task well
-- **RFT is for verifiable tasks** — math, code with test suites, structured output with exact-match graders
+- **Measure cost alongside accuracy** — compare completion tokens per response, not just accuracy scores
 - **Quality over quantity** — 200–500 high-quality examples is a good starting point
 
 ## Contributing

Skills/SKILL.md

@@ -25,6 +25,7 @@ $ARGUMENTS
 
 Read the workflow file that matches the user's current stage:
 
+- **First time / just want to get started** → `workflows/quickstart.md`
 - **Starting from scratch** → `workflows/full-pipeline.md`
 - **Need a dataset** → `workflows/dataset-creation.md`
 - **Training and iterating** → `workflows/iterative-training.md`
@@ -60,10 +61,13 @@ Reusable Python scripts in `scripts/`. Each is self-contained with inline docume
 
 | Script | Purpose |
 |--------|---------|
-| `submit_training.py` | Submit SFT or RFT jobs (SDK + REST fallback) |
+| `submit_training.py` | Submit SFT, DPO, or RFT jobs (SDK + REST fallback) |
+| `monitor_training.py` | Poll a running job until completion, streaming events in real time |
+| `calibrate_grader.py` | Run base model through your RFT grader to find optimal pass_threshold |
 | `generate_distillation_data.py` | Generate training data from a teacher model for distillation |
 | `check_training.py` | Pull training curves, detect overfitting, list checkpoints |
 | `deploy_model.py` | Deploy fine-tuned models via ARM REST API |
+| `cleanup.py` | List and delete old deployments, files, and pending jobs to reclaim quota |
 | `evaluate_model.py` | Run held-out eval with 2-dimension LLM judge |
 | `convert_dataset.py` | Convert between SFT, DPO, and RFT JSONL formats |
 | `score_dataset.py` | LLM-judge quality scoring on training data |

Skills/references/grader-design.md

@@ -138,7 +138,12 @@ for threshold in [0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95]:
     print(f"  @{threshold}: pass={pass_rate:.0%}, fail={fail_rate:.0%}")
 ```
 
-4. Choose the threshold where **30-50% of base model rollouts fail**
+4. Choose the threshold where **25-50% of base model rollouts fail**
+
+Use `scripts/calibrate_grader.py` to automate this workflow:
+```bash
+python calibrate_grader.py --model o4-mini --data train.jsonl --grader grader.py --n 30
+```
 
 | Failure Rate | Signal Quality |
 |-------------|----------------|