Skip to content

Commit 8b12a43

Browse files
iamzifeiclaude
andcommitted
feat: v0.3.0 — repositioning + task eval MVP
Major repositioning based on feedback: - Agent = orchestrator that completes tasks (Claude Code, Devin, etc.) - Tool = capability module an agent uses (MCP servers, APIs, skills) - Previous versions evaluated tools. v0.3 adds agent task evaluation. New: agent-eval task command - Define a task with success criteria in task.yaml - Execute any CLI-based agent, capture output - LLM-as-judge evaluates each criterion - Measures: success rate, duration, estimated tokens - Reports pass/fail with per-criterion reasoning Refactored: - agent-eval run → agent-eval tool (run kept as hidden alias) - PRD rewritten with agent vs tool distinction (v0.3) - Website repositioned: "AI Agent Evaluation" not just "MCP Benchmarks" - README updated with two-layer evaluation model Stats: 22 source files, 13 test files, 74 tests all passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent a7b5a10 commit 8b12a43

12 files changed

Lines changed: 965 additions & 373 deletions

File tree

README.md

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,16 @@
44
[![CI](https://github.com/OrrisTech/agent-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/OrrisTech/agent-eval/actions)
55
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
66

7-
**The credit rating agency for AI agents.** Open-source evaluation framework for benchmarking MCP servers, A2A agents, and API-first agent services.
7+
**The credit rating agency for AI agents.** Open-source evaluation platform with two layers: **Task Eval** (can the agent complete real tasks?) and **Tool Eval** (are the agent's tools reliable?).
88

99
## Quick Start
1010

1111
```bash
12-
# Initialize config for your agent
13-
npx @agenthunter/eval init
12+
# Evaluate an agent on a task
13+
npx @agenthunter/eval task -c task.yaml
1414

15-
# Run evaluation
16-
ANTHROPIC_API_KEY=your-key npx @agenthunter/eval run
17-
18-
# View results
19-
cat agent-eval-report/report.json
15+
# Benchmark an MCP server (tool quality)
16+
npx @agenthunter/eval tool -c agent-eval.yaml
2017
```
2118

2219
## What it does

0 commit comments

Comments
 (0)