Skip to content

Commit 96d0954

Browse files
iamzifeiclaude
iamzifei
and
claude
committed
feat: publish @agenthunter/eval v0.1.0 to npm + Dev.to blog post
- Published to npm as @agenthunter/eval (scoped under agenthunter org) - Usage: npx @agenthunter/eval run - Added Dev.to-formatted blog post (docs/blog/devto-mcp-benchmark.md) - Updated all docs to use correct npm package name - Added main/exports fields for library usage npm: https://www.npmjs.com/package/@agenthunter/eval Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent f786b44 commit 96d0954

4 files changed

Lines changed: 172 additions & 9 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,10 @@
66

77
```bash
88
# Initialize config for your agent
9-
npx agent-eval init
9+
npx @agenthunter/eval init
1010

1111
# Run evaluation
12-
ANTHROPIC_API_KEY=your-key npx agent-eval run
12+
ANTHROPIC_API_KEY=your-key npx @agenthunter/eval run
1313

1414
# View results
1515
cat agent-eval-report/report.json

‎docs/blog/devto-mcp-benchmark.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
title: We Benchmarked 12 MCP Servers — Here's What We Found
3+
published: false
4+
description: We built an open-source eval framework and scored 12 popular MCP servers across capability, reliability, efficiency, and safety. The results were surprising.
5+
tags: mcp, ai, agents, opensource
6+
canonical_url: https://github.com/OrrisTech/agent-eval/blob/main/docs/blog/mcp-server-benchmark.md
7+
cover_image:
8+
---
9+
10+
The MCP ecosystem has exploded. 10,000+ servers on the registry, 97 million monthly SDK downloads — but nobody can tell you which MCP server is actually worth using.
11+
12+
We decided to find out.
13+
14+
We built [agent-eval](https://github.com/OrrisTech/agent-eval), an open-source framework that automatically benchmarks MCP servers. We pointed it at 12 popular servers and scored them on 5 dimensions.
15+
16+
Some results surprised us.
17+
18+
## TL;DR Rankings
19+
20+
| Rank | Server | Score | Reliability | Category |
21+
|------|--------|-------|-------------|----------|
22+
| 🥇 | **context7** | **89** | 100% | Search |
23+
| 🥈 | **mcp-fetch** | **86** | 90% | Web |
24+
| 🥉 | **mcp-memory** | **82** | 93% | Memory |
25+
| 4 | **notion-mcp** | **82** | 97% | Productivity |
26+
| 5 | **mcp-datetime** | **81** | 73% | Utilities |
27+
| 6 | **mcp-everything** | **75** | 74% | Reference |
28+
| 7 | **mcp-sequential-thinking** | **71** | 100% | Reasoning |
29+
| 8 | **mcp-filesystem** | **68** | 14% | Filesystem |
30+
| 9 | **playwright-mcp** | **68** | 30% | Browser |
31+
| 10 | **mcp-sqlite** | **63** | 10% | Database |
32+
| 11 | **mcp-git** | **55** | 4% | DevTools |
33+
| 12 | **mcp-puppeteer** | **47** | 0% | Browser |
34+
35+
## How we tested
36+
37+
For each server, the framework:
38+
39+
1. **Connects** via stdio and discovers all available tools
40+
2. **Generates test tasks** — Claude reads each tool's JSON Schema and creates test cases (basic, edge-case, adversarial)
41+
3. **Executes tasks** multiple times to measure reliability
42+
4. **Scores output quality** using LLM-as-judge (Claude Sonnet 4)
43+
5. **Measures metrics** — latency, success rate, prompt injection resistance
44+
45+
Five dimensions, weighted:
46+
47+
| Dimension | Weight | What it measures |
48+
|-----------|--------|------------------|
49+
| **Capability** | 30% | Does the tool do what it says? |
50+
| **Reliability** | 25% | Does it work every time? |
51+
| **Efficiency** | 20% | How fast is it? |
52+
| **Safety** | 15% | Can you trick it? |
53+
| **Dev Experience** | 10% | Docs, error messages, schema quality |
54+
55+
## 3 things that surprised us
56+
57+
### 1. Reliability is the great divider
58+
59+
The gap between the best and worst is massive. **context7** and **mcp-sequential-thinking** hit 100% success rate. Meanwhile **mcp-puppeteer** scored 0% — every single task failed.
60+
61+
5 out of 12 servers couldn't even hit 50% reliability. Most failures weren't bugs — they were from tools receiving auto-generated arguments that didn't match real-world constraints (file paths that don't exist, git repos that aren't initialized).
62+
63+
This tells us something important: **MCP servers are only as reliable as the context they're given.** A filesystem server without files to read will always fail.
64+
65+
### 2. Safety is (mostly) a non-issue
66+
67+
9 out of 12 servers scored perfect 100 on safety. We tested prompt injection (malicious paths, injection attempts in arguments) and scope violations. Almost every server properly rejected out-of-scope requests.
68+
69+
The MCP protocol's design helps here — tools have typed schemas, so there's less surface for injection compared to free-text APIs.
70+
71+
### 3. Simple servers score higher
72+
73+
**context7** (2 tools, score 89) beat **notion-mcp** (22 tools, score 82). **mcp-sequential-thinking** (1 tool) scored higher than **mcp-git** (15 tools).
74+
75+
The pattern: servers that do one thing well score higher than Swiss Army knives. More tools = more surface area for failures.
76+
77+
## Try it yourself
78+
79+
```bash
80+
# Install
81+
npx @agenthunter/eval --version
82+
83+
# Create config for any MCP server
84+
cat > agent-eval.yaml << 'EOF'
85+
agent:
86+
name: "my-server"
87+
protocol: mcp
88+
endpoint: "npx -y @modelcontextprotocol/server-memory"
89+
capabilities: ["memory"]
90+
eval:
91+
runs: 3
92+
EOF
93+
94+
# Run evaluation
95+
ANTHROPIC_API_KEY=your-key npx @agenthunter/eval run
96+
```
97+
98+
Output looks like this:
99+
100+
```
101+
AgentHunter Eval v0.1.0
102+
Agent: mcp-memory v1.0.0 (MCP)
103+
Tools: 9 | Tasks: 27 | Runs: 27
104+
105+
────────────────────────────────────
106+
SCORE: 82 / 100
107+
────────────────────────────────────
108+
109+
Capability ████████████░░░░░░░░ 63%
110+
Reliability ██████████████████░░ 93%
111+
Efficiency ████████████████████ 100%
112+
Safety █████████████████░░░ 89%
113+
Dev Experience ██████████████░░░░░░ 70%
114+
```
115+
116+
## Caveats
117+
118+
- **LLM non-determinism**: Scores vary ±5 points between runs because both task generation and judging use Claude. We plan to add deterministic task sets in v0.2.
119+
- **Auto-generated tasks**: The framework generates test tasks from tool schemas. For tools that need real-world context (file systems with actual files, databases with actual data), reliability scores will be lower than real-world usage.
120+
- **DX score is a placeholder**: We score Developer Experience at a flat 70 for now. Proper DX evaluation (docs quality, error message helpfulness) is coming.
121+
- **Single model judge**: Using Claude to judge Claude-generated tasks has inherent bias. Multi-model judging is on the roadmap.
122+
123+
## What's next
124+
125+
- **A2A protocol support** — evaluate Google's Agent-to-Agent servers
126+
- **Deterministic task sets** — curated test suites per category (not just LLM-generated)
127+
- **Web dashboard** — browse rankings at eval.agenthunter.io
128+
- **Continuous monitoring** — track score changes over time
129+
130+
The framework is fully open source: **[github.com/OrrisTech/agent-eval](https://github.com/OrrisTech/agent-eval)**
131+
132+
Raw evaluation data for all 12 servers is in the [results directory](https://github.com/OrrisTech/agent-eval/tree/main/results).
133+
134+
---
135+
136+
*Built by [AgentHunter](https://agenthunter.io). We're building the quality layer for the AI agent economy — independent evaluation, transparent methodology, open data.*

‎docs/blog/mcp-server-benchmark.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,8 @@ For each server, we:
2424
All evaluation code is [open source](https://github.com/OrrisTech/agent-eval). You can reproduce these results yourself:
2525

2626
```bash
27-
npx agent-eval init
28-
npx agent-eval run
27+
npx @agenthunter/eval init
28+
npx @agenthunter/eval run
2929
```
3030

3131
## Rankings
@@ -215,7 +215,7 @@ echo 'agent:
215215
eval:
216216
runs: 3' > agent-eval.yaml
217217

218-
ANTHROPIC_API_KEY=your-key npx agent-eval run
218+
ANTHROPIC_API_KEY=your-key npx @agenthunter/eval run
219219
```
220220

221221
## What's Next

‎packages/agent-eval/package.json‎

Lines changed: 31 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,55 @@
11
{
2-
"name": "agent-eval",
2+
"name": "@agenthunter/eval",
33
"version": "0.1.0",
44
"description": "Open-source AI agent evaluation framework — benchmark MCP servers, A2A agents, and API-first agent services",
55
"type": "module",
66
"bin": {
7-
"agent-eval": "./dist/cli.js"
7+
"agent-eval": "./dist/cli.js",
8+
"agenthunter": "./dist/cli.js"
89
},
10+
"main": "./dist/index.js",
11+
"exports": {
12+
".": {
13+
"import": "./dist/index.js"
14+
}
15+
},
16+
"files": [
17+
"dist",
18+
"README.md"
19+
],
920
"scripts": {
1021
"build": "tsup",
1122
"dev": "tsup --watch",
1223
"test": "vitest run",
1324
"test:watch": "vitest",
1425
"lint": "biome check src/",
15-
"typecheck": "tsc --noEmit"
26+
"typecheck": "tsc --noEmit",
27+
"prepublishOnly": "npm run build"
1628
},
1729
"keywords": [
1830
"ai-agent",
1931
"evaluation",
2032
"benchmark",
2133
"mcp",
2234
"a2a",
23-
"agent-eval"
35+
"agent-eval",
36+
"agenthunter",
37+
"llm",
38+
"model-context-protocol"
2439
],
40+
"repository": {
41+
"type": "git",
42+
"url": "https://github.com/OrrisTech/agent-eval"
43+
},
44+
"homepage": "https://github.com/OrrisTech/agent-eval",
45+
"bugs": {
46+
"url": "https://github.com/OrrisTech/agent-eval/issues"
47+
},
48+
"author": "AgentHunter",
2549
"license": "MIT",
50+
"engines": {
51+
"node": ">=20"
52+
},
2653
"dependencies": {
2754
"@anthropic-ai/sdk": "^0.88.0",
2855
"@modelcontextprotocol/sdk": "^1.29.0",

0 commit comments

Comments
 (0)