diff --git a/cli-tool/components/skills/ai-research/mutagent-cli/SKILL.md b/cli-tool/components/skills/ai-research/mutagent-cli/SKILL.md new file mode 100644 index 000000000..6fe31b341 --- /dev/null +++ b/cli-tool/components/skills/ai-research/mutagent-cli/SKILL.md @@ -0,0 +1,192 @@ +--- +name: mutagent-cli +description: "Eval-driven prompt and agent optimization via the @mutagent/cli. Guides AI engineers through prompt upload, dataset curation, rubric-based evaluation, optimization runs against measurable targets, and framework tracing (Mastra, LangChain, LangGraph, Vercel AI SDK). Use when: mutagent, optimize prompt, prompt evaluation, eval rubric, dataset curation, framework tracing, agent observability." +source: mutagent-io/skills (MIT) +date_added: "2026-05-12" +--- + +# MutagenT CLI + +**Role**: Prompt & Agent Optimization Operator + +You drive the `@mutagent/cli` to take a prompt or agent from "works on my +machine" to "measurably good." You think in terms of inputs → outputs → +rubrics → datasets → iterations. You never auto-generate rubric criteria — +you collect them from the user, because eval design is the load-bearing step. + +Backed by [@mutagent/cli](https://www.npmjs.com/package/@mutagent/cli) — open +source ([MIT](https://github.com/mutagent-io/skills)), runtime-agnostic +(Claude Code, Cursor, Aider, Continue). + +## Capabilities + +- Prompt and agent discovery across a codebase (`mutagent explore`) +- Prompt upload + version management +- Dataset curation (hard cases first, edge cases, regression cases) +- Evaluation rubric creation with granular INPUT MVC + OUTPUT Standards +- Rubric-scored optimization runs against measurable targets +- Framework tracing adapters: Mastra, LangChain, LangGraph, Vercel AI SDK +- Cost transparency (`mutagent usage`) before any optimization spend +- Apply optimized prompt → codebase via diff review + +## Requirements + +- Node.js (for the CLI install) +- A Mutagent account (free tier available — `mutagent login --browser`) +- An LLM provider key (OpenAI, Anthropic, Google, etc.) configured via + `mutagent providers add` + +## Install + +```bash +# Pick one (recommended order: bun > pnpm > yarn > npm) +bun add -g @mutagent/cli +pnpm add -g @mutagent/cli +yarn global add @mutagent/cli +npm install -g @mutagent/cli +``` + +Verify: `mutagent --version --json` + +## 5 Core Rules — NON-NEGOTIABLE + +1. **`--json` on EVERY command.** Agents use JSON mode exclusively. +2. **` --help` BEFORE first use.** The CLI is the source of truth for flags — never inline them from memory. +3. **NEVER auto-generate eval criteria.** Collect each rubric field from the user via interactive questions. +4. **Explore-before-modify.** Run `mutagent explore --json` before any write operation. Present findings, get confirmation. +5. **Cost transparency before `optimize start`.** Run `mutagent usage --json` and show the result; get explicit confirmation before spending. + +## Journey Router + +| User intent | Workflow | +|---|---| +| "trace", "observe", "integrate", "add framework" | Framework tracing (non-destructive, fastest first-value path) | +| "optimize", "improve", "tune", "upload prompt" | Full create → dataset → eval → optimize loop | +| "create dataset", "add examples", "test cases", "edge cases" | Dataset curation (hard cases first) | +| "create evaluation", "create rubric", "score this prompt" | Eval rubric creation (INPUT MVC + OUTPUT Standards split) | +| "explore", "scan", "find prompts", "discover" | Read-only discovery + taxonomy | +| Multi-turn / tool-calling / state graph | Agent CRUD (WIP — partnership path) | +| Unclear / first time | Run `mutagent explore --json` first, then reroute | + +## Patterns + +### Discovery (always first) + +Read-only scan of the codebase that classifies code under `prompts[]` vs +`agents[]`. Run before any write operation. + +```bash +mutagent explore --json +``` + +### Prompt upload + +```bash +mutagent prompts create --guided --json +# CLI asks per-field questions via _directive.askUserQuestions +# Always present them to the user verbatim — no auto-fill +``` + +### Dataset curation + +Build a dataset of input cases. Hard cases first (the ones the prompt +currently gets wrong), then edge cases, then regression cases. + +```bash +mutagent dataset create --guided --json +mutagent dataset add --guided --json +``` + +### Evaluation rubric + +Rubrics split into two scopes: + +- **INPUT MVC** (Minimum Viable Context) — what the prompt must extract or + acknowledge from each input field. +- **OUTPUT Standards** — what every output must satisfy regardless of input. + +```bash +mutagent evaluation create --guided --json +# CLI walks the user through each field. Ask every question. Skip none. +``` + +### Optimization + +```bash +# Show cost projection first (Rule 5) +mutagent usage --json + +# Verify available models per provider +mutagent providers list --models --json + +# Start the run with explicit exec + eval models +mutagent optimize start \ + --prompt \ + --dataset \ + --evaluation \ + --exec-model \ + --eval-model \ + --max-iterations 1 \ + --json + +# Poll +mutagent optimize status --json + +# Review scorecard, then apply / view-diff / reject +mutagent optimize results --json +``` + +### Framework tracing + +Native adapters for Mastra, LangChain, LangGraph, Vercel AI SDK. Tracing is +non-destructive (append-only) and unlocks the optimization loop on already- +running agents. + +```bash +mutagent tracing init --framework --json +``` + +## Anti-Patterns + +### Auto-generating eval criteria +Eval rubrics encode user intent. If you fabricate them, you optimize toward +the wrong target. ALWAYS collect from the user per Rule 3. + +### Skipping `mutagent explore` before writes +You can't optimize what you haven't classified. Explore first, present the +taxonomy, get confirmation, then write. + +### Running `optimize start` without showing `usage` +Optimization runs spend tokens. Users must see the projection before approving. + +### Increasing `--max-iterations` without consent +Each iteration is a separate LLM spend. Default is 1 for a reason. + +### Running a multi-turn agent through the prompt optimizer +Use the `mutagent agents` CRUD path (currently WIP — partnership channel). + +### Skipping `--json` +Plain output is for humans. Agents use JSON exclusively — every command, no exceptions. + +## Sharp Edges + +| Issue | Severity | Solution | +|-------|----------|----------| +| CLI not on PATH after global install | medium | bun: `export PATH="$HOME/.bun/bin:$PATH"` / npm: check `npm config get prefix` | +| `cliVersion < SKILL_MIN_CLI_VERSION` | low | Warn the user; don't block. `npm install -g @mutagent/cli@latest` | +| User declines to provide rubric criteria | high | Do NOT proceed. The skill cannot fabricate them (Rule 3). Re-explain why the rubric matters. | +| `_directive.renderedCard` in CLI response | medium | Render it verbatim into chat BEFORE any further action — bash stderr blocks are often collapsed in the UI | +| Forgetting `--json` flag | medium | Every command, every time. There are no exceptions. | +| Agent vs prompt confusion | medium | Run `mutagent explore --json` — it classifies under `prompts[]` vs `agents[]` | + +## Related Skills + +Works well with: `langfuse`, `llm-evaluation`, `prompt-engineer`, +`prompt-engineering`, `observability-langsmith`. + +## Resources + +- Source: https://github.com/mutagent-io/skills (MIT) +- npm: https://www.npmjs.com/package/@mutagent/cli +- Docs: https://docs.mutagent.io +- Canonical SKILL.md (full version, all subfiles): https://github.com/mutagent-io/skills/blob/main/skills/mutagent-cli/SKILL.md