Skip to content

Commit d46d457

Browse files
iamzifeiclaude
andcommitted
chore: update Dev.to blog to first-person voice
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 96d0954 commit d46d457

1 file changed

Lines changed: 13 additions & 16 deletions

File tree

docs/blog/devto-mcp-benchmark.md

Lines changed: 13 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
11
---
2-
title: We Benchmarked 12 MCP Servers — Here's What We Found
2+
title: I Benchmarked 12 MCP Servers — Here's What I Found
33
published: false
4-
description: We built an open-source eval framework and scored 12 popular MCP servers across capability, reliability, efficiency, and safety. The results were surprising.
4+
description: I built an open-source eval framework and scored 12 popular MCP servers across capability, reliability, efficiency, and safety. The results were surprising.
55
tags: mcp, ai, agents, opensource
66
canonical_url: https://github.com/OrrisTech/agent-eval/blob/main/docs/blog/mcp-server-benchmark.md
77
cover_image:
88
---
99

1010
The MCP ecosystem has exploded. 10,000+ servers on the registry, 97 million monthly SDK downloads — but nobody can tell you which MCP server is actually worth using.
1111

12-
We decided to find out.
12+
I decided to find out.
1313

14-
We built [agent-eval](https://github.com/OrrisTech/agent-eval), an open-source framework that automatically benchmarks MCP servers. We pointed it at 12 popular servers and scored them on 5 dimensions.
14+
I built [agent-eval](https://github.com/OrrisTech/agent-eval), an open-source framework that automatically benchmarks MCP servers. I pointed it at 12 popular servers and scored them on 5 dimensions.
1515

16-
Some results surprised us.
16+
Some results surprised me.
1717

1818
## TL;DR Rankings
1919

@@ -32,7 +32,7 @@ Some results surprised us.
3232
| 11 | **mcp-git** | **55** | 4% | DevTools |
3333
| 12 | **mcp-puppeteer** | **47** | 0% | Browser |
3434

35-
## How we tested
35+
## How I tested
3636

3737
For each server, the framework:
3838

@@ -52,19 +52,19 @@ Five dimensions, weighted:
5252
| **Safety** | 15% | Can you trick it? |
5353
| **Dev Experience** | 10% | Docs, error messages, schema quality |
5454

55-
## 3 things that surprised us
55+
## 3 things that surprised me
5656

5757
### 1. Reliability is the great divider
5858

5959
The gap between the best and worst is massive. **context7** and **mcp-sequential-thinking** hit 100% success rate. Meanwhile **mcp-puppeteer** scored 0% — every single task failed.
6060

6161
5 out of 12 servers couldn't even hit 50% reliability. Most failures weren't bugs — they were from tools receiving auto-generated arguments that didn't match real-world constraints (file paths that don't exist, git repos that aren't initialized).
6262

63-
This tells us something important: **MCP servers are only as reliable as the context they're given.** A filesystem server without files to read will always fail.
63+
This tells me something important: **MCP servers are only as reliable as the context they're given.** A filesystem server without files to read will always fail.
6464

6565
### 2. Safety is (mostly) a non-issue
6666

67-
9 out of 12 servers scored perfect 100 on safety. We tested prompt injection (malicious paths, injection attempts in arguments) and scope violations. Almost every server properly rejected out-of-scope requests.
67+
9 out of 12 servers scored perfect 100 on safety. I tested prompt injection (malicious paths, injection attempts in arguments) and scope violations. Almost every server properly rejected out-of-scope requests.
6868

6969
The MCP protocol's design helps here — tools have typed schemas, so there's less surface for injection compared to free-text APIs.
7070

@@ -77,9 +77,6 @@ The pattern: servers that do one thing well score higher than Swiss Army knives.
7777
## Try it yourself
7878

7979
```bash
80-
# Install
81-
npx @agenthunter/eval --version
82-
8380
# Create config for any MCP server
8481
cat > agent-eval.yaml << 'EOF'
8582
agent:
@@ -115,15 +112,15 @@ Output looks like this:
115112

116113
## Caveats
117114

118-
- **LLM non-determinism**: Scores vary ±5 points between runs because both task generation and judging use Claude. We plan to add deterministic task sets in v0.2.
115+
- **LLM non-determinism**: Scores vary ±5 points between runs because both task generation and judging use Claude. Deterministic task sets are coming in v0.2.
119116
- **Auto-generated tasks**: The framework generates test tasks from tool schemas. For tools that need real-world context (file systems with actual files, databases with actual data), reliability scores will be lower than real-world usage.
120-
- **DX score is a placeholder**: We score Developer Experience at a flat 70 for now. Proper DX evaluation (docs quality, error message helpfulness) is coming.
117+
- **DX score is a placeholder**: Developer Experience is scored at a flat 70 for now. Proper DX evaluation (docs quality, error message helpfulness) is coming.
121118
- **Single model judge**: Using Claude to judge Claude-generated tasks has inherent bias. Multi-model judging is on the roadmap.
122119

123120
## What's next
124121

125122
- **A2A protocol support** — evaluate Google's Agent-to-Agent servers
126-
- **Deterministic task sets** — curated test suites per category (not just LLM-generated)
123+
- **Deterministic task sets** — curated test suites per category
127124
- **Web dashboard** — browse rankings at eval.agenthunter.io
128125
- **Continuous monitoring** — track score changes over time
129126

@@ -133,4 +130,4 @@ Raw evaluation data for all 12 servers is in the [results directory](https://git
133130

134131
---
135132

136-
*Built by [AgentHunter](https://agenthunter.io). We're building the quality layer for the AI agent economy — independent evaluation, transparent methodology, open data.*
133+
*I'm building [AgentHunter](https://agenthunter.io)the quality layer for the AI agent economy. Independent evaluation, transparent methodology, open data. If you're building an MCP server, I'd love to benchmark it — [open an issue](https://github.com/OrrisTech/agent-eval/issues).*

0 commit comments

Comments
 (0)