Add topic guide docs: generation, pipeline, verification, gates, tool contracts, production; label Claude Skill and MCP packaging as roadmap design notes

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: HelloThisWorld

README.md

@@ -218,7 +218,9 @@ npm run forge   # offline demo pipeline
 npm run clean   # remove build output, runs, and packaged zips
 ```
 
-Docs: [pipeline model](docs/pipeline-model.md) · [skill package format](docs/skill-package-format.md) · [verification integration](docs/verification-integration.md) · [model adapters](docs/model-adapters.md) · [roadmap](docs/roadmap.md)
+Reference docs: [pipeline model](docs/pipeline-model.md) · [skill package format](docs/skill-package-format.md) · [verification integration](docs/verification-integration.md) · [model adapters](docs/model-adapters.md) · [roadmap](docs/roadmap.md)
+
+Guides: [skill pipeline overview](docs/ai-agent-skill-pipeline.md) · [skill generation](docs/agent-skill-generation.md) · [verification pipeline](docs/skill-verification-pipeline.md) · [quality gates](docs/llm-eval-quality-gates.md) · [tool contracts](docs/tool-calling-skill-contracts.md) · [production practices](docs/production-agent-skills.md) · [Claude Skill packaging (roadmap)](docs/claude-skill-packaging.md) · [MCP integration (roadmap)](docs/model-context-protocol-integration.md)
 
 ## License
 

docs/agent-skill-generation.md

@@ -0,0 +1,58 @@
+# Agent skill generation
+
+How agent-skill-forge turns a requirement YAML into a complete, verifiable skill package. This covers pipeline stages 1–4; verification and release are covered in [skill-verification-pipeline.md](skill-verification-pipeline.md) and [llm-eval-quality-gates.md](llm-eval-quality-gates.md).
+
+## From requirement to normalized requirement
+
+The input is a single YAML file (see `examples/`). The schema (`src/requirements/requirement-schema.ts`) validates:
+
+- `name` (kebab-case), `version` (semver), `description`
+- `goals` (min 1), `non_goals`
+- `tools` — each with `name` (snake_case), `type` (`read` | `write`), `description`; duplicates rejected
+- `safety_requirements`, `acceptance_criteria` (min 1)
+- optional `quality_gate` thresholds
+
+Validation fails early with per-field messages. The normalizer (`requirement-normalizer.ts`) trims whitespace, converts snake_case to camelCase, and applies default quality gate thresholds (90% pass rate, 100% schema validity, 2% unsupported claims, 5% tool errors) when the file omits them. The result is persisted as `artifacts/normalized-requirement.json`.
+
+## From requirement to skill spec
+
+`generate_spec` produces an explicit behavioral contract (`artifacts/skill-spec.json`). The offline mock adapter derives it deterministically (`src/spec/skill-spec-generator.ts`):
+
+| Requirement signal | Spec consequence |
+| ------------------ | ---------------- |
+| tool `type: write` | `requiresConfirmation: true` on that tool boundary (safe default, always) |
+| safety mentions "idempoten…" | write tools get `idempotent: true` |
+| safety mentions "rate limit" | rate limit backoff/failure rules added to failure behavior |
+| safety/acceptance mention partial failure | `partial_success` reporting rule added |
+| goals/criteria mention citations or grounding | `citations` added to required output fields |
+
+Every spec declares the same five statuses (`success`, `partial_success`, `insufficient_information`, `refused`, `error`), input expectations, failure behavior, and an eval strategy (case counts scale with goals and non-goals).
+
+Real model adapters may produce richer specs, but the pipeline re-validates whatever comes back against `skillSpecSchema` (zod) at the stage boundary — an invalid spec fails the stage instead of corrupting downstream generation.
+
+## From spec to package files
+
+`generate_skill_package` renders the core files (`src/generators/`):
+
+- **SKILL.md** (`skill-md-generator.ts`) — eight fixed sections (When to Use → Acceptance Criteria). Every behavioral rule appears as concrete text: the machine-parsable `- Allowed status values:` line, per-tool confirmation and idempotency bullets, safety requirements verbatim, explicit partial-failure and missing-field handling.
+- **skill.manifest.json** (`manifest-generator.ts`) — name, version, `generatedBy`, `generatedAt`, tool list, file map, quality gate.
+- **tools.json** (`tools-generator.ts`) — one contract per tool with a JSON-Schema-style `inputSchema`; confirmation-gated tools require `user_confirmed`, idempotent tools require `idempotency_key`. Details in [tool-calling-skill-contracts.md](tool-calling-skill-contracts.md).
+- **examples.md** — happy path, missing-information edge case, refusal, and (when a write tool exists) a confirmed-write example with real JSON outputs.
+- **README.md** — package-level overview and regeneration instructions.
+
+## From spec to eval cases
+
+`generate_eval_cases` (`eval-case-generator.ts`) emits machine-checkable cases:
+
+- **golden** — one per goal; expect `success`, check `structured_output`, `no_unsupported_claims` (plus `mentions:citation` for grounded skills)
+- **negative** — one per non-goal; expect `refused`, check the non-goal is covered verbatim
+- **edge** — missing required fields (`insufficient_information`), partial workflow success (`partial_success`, when relevant), out-of-scope request (`refused`)
+- **tool_failure** — rate-limit failure and confirmation-bypass attempts against write tools
+
+Each case carries an expected status plus capability checks (`status:…`, `confirm:…`, `mentions:…`; full grammar in [skill-package-format.md](skill-package-format.md)). Verification rules (`verification-rules-generator.ts`) — required files, sections, statuses, and forbidden claim patterns like "guarantee" or "never fails" — complete the package.
+
+## Properties worth relying on
+
+- **Deterministic**: with `--model mock`, the same YAML always yields the same package (timestamps aside). CI and tests depend on this.
+- **Self-describing**: the package embeds its own eval cases and verification rules, so any conforming verifier can re-check it later without the original requirement.
+- **Regenerable**: the requirement file is the source of truth. To change a skill, edit the YAML and re-run the pipeline; hand-edits to generated files are overwritten (see [production-agent-skills.md](production-agent-skills.md)).

docs/ai-agent-skill-pipeline.md

@@ -0,0 +1,51 @@
+# The AI agent skill pipeline
+
+Why treat agent skill creation as a pipeline, and how agent-skill-forge implements one. This is the conceptual overview; [pipeline-model.md](pipeline-model.md) is the stage-by-stage reference.
+
+## The problem with prompt-and-pray
+
+An agent skill is usually born as a prompt file: someone writes instructions, reads them over, and ships. There is no spec to diff against, no eval suite to catch regressions, no record of why a version was considered good, and no gate between "edited the prompt" and "agents now behave differently in production."
+
+Regular software solved this decades ago: requirements → build → test → release, with artifacts and gates at each step. agent-skill-forge applies the same shape to skills.
+
+## The pipeline
+
+```
+requirement (YAML)                        author intent, versioned in git
+  -> normalized requirement               validated, defaulted, canonical
+  -> skill spec                           explicit behavioral contract
+  -> skill package                        SKILL.md + manifest + tool contracts
+  -> eval cases + verification rules      machine-checkable expectations
+  -> verification                         metrics: pass / schema / claims / tool errors
+  -> failure analysis                     typed failure categories
+  -> repair loop                          spec-driven fixes, re-verified
+  -> quality gate                         release decision
+  -> packaged verified skill (.zip)       with provenance and evidence
+```
+
+Nine stages, executed by a lightweight skill-specific runner (`src/core/pipeline-runner.ts`) — deliberately not a general-purpose CI system. Stages share a typed `PipelineState`, persist every intermediate to disk, and re-validate model outputs with zod at each boundary.
+
+## What each phase buys you
+
+**Authoring** (stages 1–2). The requirement file is the source of truth: goals, non-goals, tools, safety requirements, acceptance criteria, and the quality bar itself. The spec makes implicit intent explicit — which tools need confirmation, what the output contract is, how failures must be reported.
+
+**Generation** (stages 3–4). The skill package and its eval suite are generated *together from the same spec*, so the tests always encode the current intent. See [agent-skill-generation.md](agent-skill-generation.md).
+
+**Verification** (stages 5–6). Every package faces the same checks regardless of which model generated it. Failures are classified into typed categories (`missing_confirmation`, `partial_failure_not_handled`, `unsupported_claim`, …) rather than a wall of red. See [skill-verification-pipeline.md](skill-verification-pipeline.md).
+
+**Repair** (stages 7–8). Failure analysis drives a bounded repair loop (default 2 attempts). A guard makes cheating structurally impossible: repairs may not touch `eval-cases.json` or `verification-rules.json`. Before/after metrics are recorded so a regression is visible, not silent.
+
+**Release** (stage 9). The quality gate is the only path to a package. A failed gate produces a failure summary and exit code 1 — never a zip. See [llm-eval-quality-gates.md](llm-eval-quality-gates.md).
+
+## Model-agnostic by construction
+
+Generation goes through the `ModelAdapter` interface ([model-adapters.md](model-adapters.md)). Different models may write different skills; none of them get a different gate. The offline `mock` adapter keeps the whole pipeline runnable in CI with no API keys, and `mock-flaky` exists purely to prove the failure → analysis → repair → pass loop works end to end:
+
+```bash
+npm run forge          # deterministic pass: 11/11 cases, gate PASSED, zip produced
+npm run forge:flaky    # 27% pass rate, 6 failure categories, 1 repair, then 100%
+```
+
+## Where this pipeline ends
+
+The pipeline verifies structure and declared behavior offline. It does not execute the skill against a live model (the default verifier is demo-level by design), and generated skills still require human review before production use. Those boundaries, and the plan to push them, are documented in [production-agent-skills.md](production-agent-skills.md) and [roadmap.md](roadmap.md).

docs/claude-skill-packaging.md

@@ -0,0 +1,56 @@
+# Claude Skill packaging
+
+> **Status: roadmap.** A dedicated Claude Skill packaging target is planned but not implemented. This page documents how today's generated package maps onto Claude's Agent Skills conventions, a manual conversion recipe that works now, and the planned automation. Nothing here should be read as a shipped `--target claude-skill` flag.
+
+## What Claude expects
+
+Claude Code and the Claude Agent SDK discover skills as a directory containing a `SKILL.md` whose body holds the instructions and whose YAML frontmatter carries at least a `name` and a `description` (the description drives skill triggering). Project skills live under `.claude/skills/<skill-name>/`, personal skills under `~/.claude/skills/<skill-name>/`, and plugins can bundle skills as well. Supporting files may sit next to `SKILL.md`.
+
+## What forge generates today
+
+A forge package (see [skill-package-format.md](skill-package-format.md)) is already a flat directory centered on a `SKILL.md`:
+
+```
+calendar-scheduling-skill-1.0.0.zip
+├── SKILL.md                 # instructions — no YAML frontmatter yet
+├── skill.manifest.json      # name, version, description, tools, quality gate
+├── tools.json               # tool contracts
+├── eval-cases.json          # verification evidence
+├── verification-rules.json
+├── examples.md
+├── README.md
+├── verification-report.html # gate evidence
+├── verification-summary.json
+└── package-info.json        # provenance: run ID, model, verifier, metrics
+```
+
+The overlap is deliberate: the SKILL.md body sections (When to Use, When Not to Use, Tool Use Rules, Safety Rules, Failure Behavior, …) are exactly the content a Claude skill wants. The gap is the frontmatter — forge keeps `name`/`description` in `skill.manifest.json` instead.
+
+## Manual conversion (works today)
+
+1. Unzip the release package.
+2. Prepend frontmatter to `SKILL.md`, copying the fields from `skill.manifest.json`:
+
+   ```markdown
+   ---
+   name: calendar-scheduling-skill
+   description: Help an agent schedule calendar events safely. Use when the user asks to check availability, propose meeting slots, or create calendar events.
+   ---
+
+   # calendar-scheduling-skill
+   ...existing generated body...
+   ```
+
+   Tip: extend the manifest description with "Use when …" trigger phrasing — the frontmatter description is what the model reads when deciding to activate the skill.
+3. Copy the folder to `.claude/skills/calendar-scheduling-skill/`. Keep `examples.md` (useful context); the eval/verification files are optional at runtime — they are evidence, not instructions — but keeping them documents why this skill version was released.
+4. Review the skill by hand before use. Forge's verification is structural; it does not execute the skill against a live model.
+
+## Planned automation (roadmap)
+
+Tracked in [roadmap.md](roadmap.md) ("Claude Skill packaging examples"):
+
+- A packaging target (e.g. `--package-format claude-skill`) that emits the frontmatter from the manifest, writes the `.claude/skills/<name>/` layout, and separates runtime files from verification evidence.
+- Trigger-description generation: derive the "Use when …" sentence from the requirement's goals so activation quality is part of the spec, not an afterthought.
+- A verification rule that lints the frontmatter (name matches manifest, description length, no forbidden claims) so the same quality gate covers the Claude-specific surface.
+
+If you want this sooner, the seam to extend is `src/packaging/package-builder.ts` — packaging is a pure function over the verified skill directory.

docs/llm-eval-quality-gates.md

@@ -0,0 +1,72 @@
+# LLM eval quality gates
+
+The quality gate is the release decision: a skill package ships only when its verification metrics clear the thresholds the requirement itself declares. Implementation: `src/core/quality-gate.ts`.
+
+## The four metrics
+
+All four are computed per verification run over the eval case results:
+
+| Metric | Definition (local-demo verifier) | Gate rule |
+| ------ | -------------------------------- | --------- |
+| `passRate` | cases whose checks all pass and whose expected status is declared / total cases | `>= min_pass_rate` |
+| `schemaValidRate` | cases whose expected status is part of the declared output contract / total | `>= min_schema_valid_rate` |
+| `unsupportedClaimRate` | cases flagged by a failed `no_unsupported_claims` check / total | `<= max_unsupported_claim_rate` |
+| `toolErrorRate` | failed `tool_failure`-category cases / total | `<= max_tool_error_rate` |
+
+On top of the metric rules, **any failed static check fails the gate** — a missing file, invalid manifest, or unconfirmed write tool is unreleasable regardless of pass rate. Each violation becomes a human-readable reason string in the gate result:
+
+```json
+{
+  "passed": false,
+  "reasons": [
+    "pass rate 27.3% is below minimum 90.0%",
+    "schema valid rate 90.9% is below minimum 100.0%",
+    "static check failed: write_tools_confirmation (All write tools require explicit user confirmation)"
+  ],
+  "metrics": { "passRate": 0.273, "schemaValidRate": 0.909, "unsupportedClaimRate": 0.364, "toolErrorRate": 0.182 }
+}
+```
+
+## Thresholds live in the requirement
+
+The skill requirement YAML owns its quality bar:
+
+```yaml
+quality_gate:
+  min_pass_rate: 0.90
+  min_schema_valid_rate: 1.00
+  max_unsupported_claim_rate: 0.02
+  max_tool_error_rate: 0.05
+```
+
+Omit the block and the defaults above apply (`src/config/default-config.ts`). Guidance: keep `min_schema_valid_rate` at 1.00 — a skill that can emit undeclared statuses breaks every consumer; loosen `min_pass_rate` only for exploratory skills, and say why in the requirement file, which is versioned in git.
+
+## What a failed gate does
+
+- `package_if_passed` records a **failed** stage with the gate reasons; no zip is written to the run dir or `dist/`.
+- The terminal summary prints `Result: FAILED` plus each reason; the HTML report shows thresholds vs. actuals side by side.
+- The CLI exits `1` (gate failed) vs. `0` (passed) vs. `2` (configuration/pipeline error) — pipe-friendly for CI.
+
+`--allow-failed-package` overrides the block for debugging. It is labeled UNSAFE, the stage note says the gate failed, and the run still reports FAILED — the override produces an artifact, never a green build.
+
+## Gates cannot be gamed from inside the loop
+
+Two structural guarantees keep the gate honest:
+
+1. **Repairs may not touch the tests.** `src/repair/skill-repairer.ts` rejects any adapter output that modifies `eval-cases.json` or `verification-rules.json` and logs the rejection into the repair changes.
+2. **Thresholds are read from the requirement, not from the repaired package.** A repair that edited the manifest's quality gate would change nothing — the gate evaluates against the spec's thresholds.
+
+The only legitimate ways to pass are improving the skill or consciously editing the requirement (which is a reviewable git diff).
+
+## Using the gate in CI
+
+The repo's own workflow (`.github/workflows/ci.yml`) is the reference pattern:
+
+```yaml
+- run: npm run forge
+- run: |
+    test -f runs/latest/reports/summary.json
+    node -e "const s=require('./runs/latest/reports/summary.json'); if(!s.passed) process.exit(1)"
+```
+
+Because the default model and verifier are offline and deterministic, the gate produces the same verdict on every machine — no flaky API calls in the release path. When real model adapters land ([roadmap.md](roadmap.md)), the recommended pattern is unchanged: generation may be stochastic, but the gate stays deterministic over whatever was generated.