You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/blog/devto-mcp-benchmark.md
+13-16Lines changed: 13 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,19 +1,19 @@
1
1
---
2
-
title: We Benchmarked 12 MCP Servers — Here's What We Found
2
+
title: I Benchmarked 12 MCP Servers — Here's What I Found
3
3
published: false
4
-
description: We built an open-source eval framework and scored 12 popular MCP servers across capability, reliability, efficiency, and safety. The results were surprising.
4
+
description: I built an open-source eval framework and scored 12 popular MCP servers across capability, reliability, efficiency, and safety. The results were surprising.
The MCP ecosystem has exploded. 10,000+ servers on the registry, 97 million monthly SDK downloads — but nobody can tell you which MCP server is actually worth using.
11
11
12
-
We decided to find out.
12
+
I decided to find out.
13
13
14
-
We built [agent-eval](https://github.com/OrrisTech/agent-eval), an open-source framework that automatically benchmarks MCP servers. We pointed it at 12 popular servers and scored them on 5 dimensions.
14
+
I built [agent-eval](https://github.com/OrrisTech/agent-eval), an open-source framework that automatically benchmarks MCP servers. I pointed it at 12 popular servers and scored them on 5 dimensions.
The gap between the best and worst is massive. **context7** and **mcp-sequential-thinking** hit 100% success rate. Meanwhile **mcp-puppeteer** scored 0% — every single task failed.
60
60
61
61
5 out of 12 servers couldn't even hit 50% reliability. Most failures weren't bugs — they were from tools receiving auto-generated arguments that didn't match real-world constraints (file paths that don't exist, git repos that aren't initialized).
62
62
63
-
This tells us something important: **MCP servers are only as reliable as the context they're given.** A filesystem server without files to read will always fail.
63
+
This tells me something important: **MCP servers are only as reliable as the context they're given.** A filesystem server without files to read will always fail.
64
64
65
65
### 2. Safety is (mostly) a non-issue
66
66
67
-
9 out of 12 servers scored perfect 100 on safety. We tested prompt injection (malicious paths, injection attempts in arguments) and scope violations. Almost every server properly rejected out-of-scope requests.
67
+
9 out of 12 servers scored perfect 100 on safety. I tested prompt injection (malicious paths, injection attempts in arguments) and scope violations. Almost every server properly rejected out-of-scope requests.
68
68
69
69
The MCP protocol's design helps here — tools have typed schemas, so there's less surface for injection compared to free-text APIs.
70
70
@@ -77,9 +77,6 @@ The pattern: servers that do one thing well score higher than Swiss Army knives.
77
77
## Try it yourself
78
78
79
79
```bash
80
-
# Install
81
-
npx @agenthunter/eval --version
82
-
83
80
# Create config for any MCP server
84
81
cat > agent-eval.yaml << 'EOF'
85
82
agent:
@@ -115,15 +112,15 @@ Output looks like this:
115
112
116
113
## Caveats
117
114
118
-
-**LLM non-determinism**: Scores vary ±5 points between runs because both task generation and judging use Claude. We plan to add deterministic task sets in v0.2.
115
+
-**LLM non-determinism**: Scores vary ±5 points between runs because both task generation and judging use Claude. Deterministic task sets are coming in v0.2.
119
116
-**Auto-generated tasks**: The framework generates test tasks from tool schemas. For tools that need real-world context (file systems with actual files, databases with actual data), reliability scores will be lower than real-world usage.
120
-
-**DX score is a placeholder**: We score Developer Experience at a flat 70 for now. Proper DX evaluation (docs quality, error message helpfulness) is coming.
117
+
-**DX score is a placeholder**: Developer Experience is scored at a flat 70 for now. Proper DX evaluation (docs quality, error message helpfulness) is coming.
121
118
-**Single model judge**: Using Claude to judge Claude-generated tasks has inherent bias. Multi-model judging is on the roadmap.
-**Deterministic task sets** — curated test suites per category (not just LLM-generated)
123
+
-**Deterministic task sets** — curated test suites per category
127
124
-**Web dashboard** — browse rankings at eval.agenthunter.io
128
125
-**Continuous monitoring** — track score changes over time
129
126
@@ -133,4 +130,4 @@ Raw evaluation data for all 12 servers is in the [results directory](https://git
133
130
134
131
---
135
132
136
-
*Built by[AgentHunter](https://agenthunter.io). We're building the quality layer for the AI agent economy — independent evaluation, transparent methodology, open data.*
133
+
*I'm building[AgentHunter](https://agenthunter.io) — the quality layer for the AI agent economy. Independent evaluation, transparent methodology, open data. If you're building an MCP server, I'd love to benchmark it — [open an issue](https://github.com/OrrisTech/agent-eval/issues).*
0 commit comments