Skip to content

agent-skill-generation.md

Latest commit

 

History

History
58 lines (39 loc) · 4.84 KB

agent-skill-generation.md

File metadata and controls

58 lines (39 loc) · 4.84 KB
Raw

Agent skill generation

How agent-skill-forge turns a requirement YAML into a complete, verifiable skill package. This covers pipeline stages 1–4; verification and release are covered in skill-verification-pipeline.md and llm-eval-quality-gates.md.

From requirement to normalized requirement

The input is a single YAML file (see examples/). The schema (src/requirements/requirement-schema.ts) validates:

  • name (kebab-case), version (semver), description
  • goals (min 1), non_goals
  • tools — each with name (snake_case), type (read | write), description; duplicates rejected
  • safety_requirements, acceptance_criteria (min 1)
  • optional quality_gate thresholds

Validation fails early with per-field messages. The normalizer (requirement-normalizer.ts) trims whitespace, converts snake_case to camelCase, and applies default quality gate thresholds (90% pass rate, 100% schema validity, 2% unsupported claims, 5% tool errors) when the file omits them. The result is persisted as artifacts/normalized-requirement.json.

From requirement to skill spec

generate_spec produces an explicit behavioral contract (artifacts/skill-spec.json). The offline mock adapter derives it deterministically (src/spec/skill-spec-generator.ts):

Requirement signal Spec consequence
tool type: write requiresConfirmation: true on that tool boundary (safe default, always)
safety mentions "idempoten…" write tools get idempotent: true
safety mentions "rate limit" rate limit backoff/failure rules added to failure behavior
safety/acceptance mention partial failure partial_success reporting rule added
goals/criteria mention citations or grounding citations added to required output fields

Every spec declares the same five statuses (success, partial_success, insufficient_information, refused, error), input expectations, failure behavior, and an eval strategy (case counts scale with goals and non-goals).

Real model adapters may produce richer specs, but the pipeline re-validates whatever comes back against skillSpecSchema (zod) at the stage boundary — an invalid spec fails the stage instead of corrupting downstream generation.

From spec to package files

generate_skill_package renders the core files (src/generators/):

  • SKILL.md (skill-md-generator.ts) — eight fixed sections (When to Use → Acceptance Criteria). Every behavioral rule appears as concrete text: the machine-parsable - Allowed status values: line, per-tool confirmation and idempotency bullets, safety requirements verbatim, explicit partial-failure and missing-field handling.
  • skill.manifest.json (manifest-generator.ts) — name, version, generatedBy, generatedAt, tool list, file map, quality gate.
  • tools.json (tools-generator.ts) — one contract per tool with a JSON-Schema-style inputSchema; confirmation-gated tools require user_confirmed, idempotent tools require idempotency_key. Details in tool-calling-skill-contracts.md.
  • examples.md — happy path, missing-information edge case, refusal, and (when a write tool exists) a confirmed-write example with real JSON outputs.
  • README.md — package-level overview and regeneration instructions.

From spec to eval cases

generate_eval_cases (eval-case-generator.ts) emits machine-checkable cases:

  • golden — one per goal; expect success, check structured_output, no_unsupported_claims (plus mentions:citation for grounded skills)
  • negative — one per non-goal; expect refused, check the non-goal is covered verbatim
  • edge — missing required fields (insufficient_information), partial workflow success (partial_success, when relevant), out-of-scope request (refused)
  • tool_failure — rate-limit failure and confirmation-bypass attempts against write tools

Each case carries an expected status plus capability checks (status:…, confirm:…, mentions:…; full grammar in skill-package-format.md). Verification rules (verification-rules-generator.ts) — required files, sections, statuses, and forbidden claim patterns like "guarantee" or "never fails" — complete the package.

Properties worth relying on

  • Deterministic: with --model mock, the same YAML always yields the same package (timestamps aside). CI and tests depend on this.
  • Self-describing: the package embeds its own eval cases and verification rules, so any conforming verifier can re-check it later without the original requirement.
  • Regenerable: the requirement file is the source of truth. To change a skill, edit the YAML and re-run the pipeline; hand-edits to generated files are overwritten (see production-agent-skills.md).