Commit 8b12a43
feat: v0.3.0 — repositioning + task eval MVP
Major repositioning based on feedback:
- Agent = orchestrator that completes tasks (Claude Code, Devin, etc.)
- Tool = capability module an agent uses (MCP servers, APIs, skills)
- Previous versions evaluated tools. v0.3 adds agent task evaluation.
New: agent-eval task command
- Define a task with success criteria in task.yaml
- Execute any CLI-based agent, capture output
- LLM-as-judge evaluates each criterion
- Measures: success rate, duration, estimated tokens
- Reports pass/fail with per-criterion reasoning
Refactored:
- agent-eval run → agent-eval tool (run kept as hidden alias)
- PRD rewritten with agent vs tool distinction (v0.3)
- Website repositioned: "AI Agent Evaluation" not just "MCP Benchmarks"
- README updated with two-layer evaluation model
Stats: 22 source files, 13 test files, 74 tests all passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent a7b5a10 commit 8b12a43
12 files changed
Lines changed: 965 additions & 373 deletions
File tree
- docs
- packages
- agent-eval
- src
- commands
- config
- eval
- tests
- web/src/app
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
| 12 | + | |
| 13 | + | |
14 | 14 | | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
| 15 | + | |
| 16 | + | |
20 | 17 | | |
21 | 18 | | |
22 | 19 | | |
| |||
0 commit comments