How a generated skill package is checked before it can be released. Adapter wiring and the optional external framework bridge are covered in verification-integration.md; this page explains what verification actually computes and what evidence it leaves behind.
Verification consumes the skill package directory plus the normalized requirement and spec, and produces a VerificationResult:
- static checks — package-level structural findings
- case results — one per eval case, with
passed,schemaValid,unsupportedClaim,toolError, failed checks, and a typed failure category - metrics —
passRate,schemaValidRate,unsupportedClaimRate,toolErrorRate
The pipeline validates the result against a zod schema (verification-result-schema.ts), so a misbehaving adapter fails loudly at the stage boundary.
The default local-demo verifier (src/verification/local-demo-verifier.ts) first checks the package as an artifact:
- every required file exists (from
verification-rules.json) skill.manifest.json,tools.json,eval-cases.json,verification-rules.jsonparse and satisfy their schemas- every tool from the requirement is defined in
tools.json - write tools are confirmation-gated when the rules demand it
- eval cases cover golden, negative, and edge categories
- SKILL.md contains the required sections and declares every required status
- SKILL.md contains no forbidden claim patterns ("guarantee", "never fails", …)
Any failed static check fails the quality gate outright — a package with a missing file or an unconfirmed write tool is never releasable, whatever its pass rate.
Each eval case carries machine-readable checks (grammar in skill-package-format.md). The verifier evaluates them against the package content: status:partial_success passes only if the SKILL.md output contract declares that status; confirm:create_event passes only if tools.json gates the tool and SKILL.md states the confirmation rule; no_unsupported_claims scans against the forbidden patterns.
A case passes when all its checks pass and its expected status is part of the declared output contract. Failed checks map to typed categories — missing_confirmation, partial_failure_not_handled, rate_limit_not_handled, unsupported_claim, invalid_tool_contract, unclear_skill_boundary, and so on — which is what makes the downstream repair loop targeted rather than guesswork.
Honest scope note: this simulates verification against the package's declared behavior. It does not run the skill against a live model; that depth belongs to external frameworks behind the adapter interface.
Every verification run persists:
| Artifact | Content |
|---|---|
artifacts/verification-summary.json |
full result of the latest run |
artifacts/verification-summary-initial.json |
frozen first-run result, for before/after comparison |
artifacts/verification-events.jsonl |
append-only event stream: verification_started, one static_check and case_result per check/case, verification_completed with metrics |
replay-artifacts/<label>-<caseId>.json |
for every failed case: the full case input, expectation, and observed result |
reports/verification-report.html |
self-contained human-readable report, also embedded in the release zip |
When the gate fails and repair attempts remain, the pipeline repairs and re-runs verification on the repaired directory. The rerun_verification stage records the pass-rate delta and labels it improvement, no change, or REGRESSION — a repair that makes things worse is visible in the stage timeline and the report, not hidden by the retry.
analyze_failures groups failed cases and failed static checks by category, counts them, keeps example reasons, and attaches a suggested repair per category (src/repair/failure-analyzer.ts). This grouping is the contract between verification and repair: the repair planner maps categories to target files, never to individual test tweaks.