fix: 9/12 MCP servers evaluated successfully with in-process engine

Results after fixing the execFileSync connection issue:
- context7: 89/100 (100% reliability, 2 tools) — top performer
- mcp-memory: 82/100 (93% reliability, 9 tools)
- notion-mcp: 82/100 (97% reliability, 22 tools) — most tools
- mcp-datetime: 81/100 (73% reliability, 10 tools)
- mcp-everything: 75/100 (74% reliability, 13 tools)
- mcp-sequential-thinking: 71/100 (100% reliability, 1 tool)
- mcp-filesystem: 68/100 (14% reliability, 14 tools)
- mcp-git: 55/100 (4% reliability, 15 tools)
- mcp-puppeteer: 47/100 (0% reliability, 7 tools)

3 servers still failing: mcp-fetch (timeout), mcp-sqlite (connection
closed), playwright-mcp (too slow for batch).

Blog post and summary regenerated with full 9-server rankings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: iamzifei

docs/blog/mcp-server-benchmark.md

@@ -1,10 +1,10 @@
 ---
-title: "We Benchmarked 5 MCP Servers — Here's What We Found"
+title: "We Benchmarked 9 MCP Servers — Here's What We Found"
 date: 2026-04-13
 author: AgentHunter Eval
 ---
 
-# We Benchmarked 5 MCP Servers — Here's What We Found
+# We Benchmarked 9 MCP Servers — Here's What We Found
 
 The Model Context Protocol (MCP) ecosystem has exploded — over 10,000 servers on the official registry, 97 million monthly SDK downloads. But which MCP servers are actually good?
 
@@ -32,114 +32,132 @@ npx agent-eval run
 
 | Rank | Server | Category | Score | Capability | Reliability | Efficiency | Safety |
 |------|--------|----------|-------|------------|-------------|------------|--------|
-| 1 🥇 | **mcp-sequential-thinking** | Reasoning | **77** | 33 | 100 | 100 | 100 |
-| 2 🥈 | **mcp-datetime** | Utilities | **75** | 52 | 73 | 100 | 92 |
-| 3 🥉 | **context7** | Search | **73** | 44 | 100 | 88 | 67 |
-| 4 | **mcp-everything** | Reference | **68** | 48 | 77 | 76 | 82 |
-| 5 | **mcp-filesystem** | Filesystem | **62** | 54 | 14 | 100 | 100 |
+| 1 🥇 | **context7** | Search | **89** | 83 | 100 | 87 | 100 |
+| 2 🥈 | **mcp-memory** | Memory | **82** | 63 | 93 | 100 | 89 |
+| 3 🥉 | **notion-mcp** | Productivity | **82** | 55 | 97 | 98 | 100 |
+| 4 | **mcp-datetime** | Utilities | **81** | 70 | 73 | 100 | 100 |
+| 5 | **mcp-everything** | Reference | **75** | 66 | 74 | 78 | 97 |
+| 6 | **mcp-sequential-thinking** | Reasoning | **71** | 15 | 100 | 100 | 100 |
+| 7 | **mcp-filesystem** | Filesystem | **68** | 73 | 14 | 100 | 100 |
+| 8 | **mcp-git** | DevTools | **55** | 40 | 4 | 100 | 98 |
+| 9 | **mcp-puppeteer** | Browser | **47** | 51 | 0 | 50 | 100 |
 
 
 ## Key Findings
 
 ### 1. Reliability varies wildly
 
-Of 5 servers tested, 2 achieved 80%+ reliability. However, 1 server(s) fell below 50%: **mcp-filesystem** (14%). Low reliability usually means the server crashes, times out, or returns errors for valid inputs.
+Of 9 servers tested, 4 achieved 80%+ reliability. However, 3 server(s) fell below 50%: **mcp-filesystem** (14%), **mcp-git** (4%), **mcp-puppeteer** (0%). Low reliability usually means the server crashes, times out, or returns errors for valid inputs.
 
 ### 2. Efficiency is generally excellent
 
-Average latency across all servers was 879ms. 3/5 servers scored 90+ on efficiency, meaning sub-second response times. MCP's stdio transport is inherently fast since there's no network overhead.
+Average latency across all servers was 560ms. 6/9 servers scored 90+ on efficiency, meaning sub-second response times. MCP's stdio transport is inherently fast since there's no network overhead.
 
 ### 3. Safety scores reveal gaps
 
-2/5 servers scored a perfect 100 on safety. 1 scored below 70: **context7** (67). Lower safety scores indicate the server may be susceptible to prompt injection or may expose data outside its intended scope.
+6/9 servers scored a perfect 100 on safety. 
 
 ## Individual Results
 
-### mcp-sequential-thinking
+### context7
 
-- **Category**: Reasoning
-- **Score**: 77/100
-- **Tools discovered**: 1
-- **Tasks generated**: 3
+- **Category**: Search
+- **Score**: 89/100
+- **Tools discovered**: 2
+- **Tasks generated**: 4
 - **Success rate**: 100%
-- **Avg latency**: 2ms
-- **Breakdown**: Cap 33 | Rel 100 | Eff 100 | Safe 100 | DX 70
+- **Avg latency**: 1756ms
+- **Breakdown**: Cap 83 | Rel 100 | Eff 87 | Safe 100 | DX 70
+
+### mcp-memory
+
+- **Category**: Memory
+- **Score**: 82/100
+- **Tools discovered**: 9
+- **Tasks generated**: 27
+- **Success rate**: 93%
+- **Avg latency**: 1ms
+- **Breakdown**: Cap 63 | Rel 93 | Eff 100 | Safe 89 | DX 70
+
+### notion-mcp
+
+- **Category**: Productivity
+- **Score**: 82/100
+- **Tools discovered**: 22
+- **Tasks generated**: 44
+- **Success rate**: 97%
+- **Avg latency**: 643ms
+- **Breakdown**: Cap 55 | Rel 97 | Eff 98 | Safe 100 | DX 70
 
 ### mcp-datetime
 
 - **Category**: Utilities
-- **Score**: 75/100
+- **Score**: 81/100
 - **Tools discovered**: 10
 - **Tasks generated**: 30
 - **Success rate**: 73%
 - **Avg latency**: 2ms
-- **Breakdown**: Cap 52 | Rel 73 | Eff 100 | Safe 92 | DX 70
-
-### context7
-
-- **Category**: Search
-- **Score**: 73/100
-- **Tools discovered**: 2
-- **Tasks generated**: 6
-- **Success rate**: 100%
-- **Avg latency**: 1642ms
-- **Breakdown**: Cap 44 | Rel 100 | Eff 88 | Safe 67 | DX 70
+- **Breakdown**: Cap 70 | Rel 73 | Eff 100 | Safe 100 | DX 70
 
 ### mcp-everything
 
 - **Category**: Reference
-- **Score**: 68/100
+- **Score**: 75/100
 - **Tools discovered**: 13
 - **Tasks generated**: 39
-- **Success rate**: 77%
-- **Avg latency**: 2747ms
-- **Breakdown**: Cap 48 | Rel 77 | Eff 76 | Safe 82 | DX 70
+- **Success rate**: 74%
+- **Avg latency**: 2621ms
+- **Breakdown**: Cap 66 | Rel 74 | Eff 78 | Safe 97 | DX 70
+
+### mcp-sequential-thinking
+
+- **Category**: Reasoning
+- **Score**: 71/100
+- **Tools discovered**: 1
+- **Tasks generated**: 3
+- **Success rate**: 100%
+- **Avg latency**: 1ms
+- **Breakdown**: Cap 15 | Rel 100 | Eff 100 | Safe 100 | DX 70
 
 ### mcp-filesystem
 
 - **Category**: Filesystem
-- **Score**: 62/100
+- **Score**: 68/100
 - **Tools discovered**: 14
-- **Tasks generated**: 42
+- **Tasks generated**: 28
 - **Success rate**: 14%
 - **Avg latency**: 1ms
-- **Breakdown**: Cap 54 | Rel 14 | Eff 100 | Safe 100 | DX 70
+- **Breakdown**: Cap 73 | Rel 14 | Eff 100 | Safe 100 | DX 70
+
+### mcp-git
+
+- **Category**: DevTools
+- **Score**: 55/100
+- **Tools discovered**: 15
+- **Tasks generated**: 45
+- **Success rate**: 4%
+- **Avg latency**: 18ms
+- **Breakdown**: Cap 40 | Rel 4 | Eff 100 | Safe 98 | DX 70
+
+### mcp-puppeteer
+
+- **Category**: Browser
+- **Score**: 47/100
+- **Tools discovered**: 7
+- **Tasks generated**: 14
+- **Success rate**: 0%
+- **Avg latency**: 0ms
+- **Breakdown**: Cap 51 | Rel 0 | Eff 50 | Safe 100 | DX 70
 
 
 
 ## Servers That Failed Evaluation
 
 These servers could not be evaluated (connection failures, crashes, or missing dependencies):
 
-- **mcp-memory** (Memory): - Loading config...
-✔ Config loaded: mcp-memory (MCP)
-- Connecting to agent via MCP...
-[3
-- **mcp-fetch** (Web): - Loading config...
-✔ Config loaded: mcp-fetch (MCP)
-- Connecting to agent via MCP...
-
-Err
-- **mcp-git** (DevTools): - Loading config...
-✔ Config loaded: mcp-git (MCP)
-- Connecting to agent via MCP...
-✔
-- **playwright-mcp** (Browser): - Loading config...
-✔ Config loaded: playwright-mcp (MCP)
-- Connecting to agent via MCP...
-- **mcp-puppeteer** (Browser): - Loading config...
-✔ Config loaded: mcp-puppeteer (MCP)
-- Connecting to agent via MCP...
-
-- **mcp-sqlite** (Database): - Loading config...
-✔ Config loaded: mcp-sqlite (MCP)
-- Connecting to agent via MCP...
-
-Er
-- **notion-mcp** (Productivity): - Loading config...
-✔ Config loaded: notion-mcp (MCP)
-- Connecting to agent via MCP...
-[3
+- **mcp-fetch** (Web): No report generated
+- **mcp-sqlite** (Database): No report generated
+- **playwright-mcp** (Browser): No report generated
 
 
 

results/context7/agent-eval.yaml

@@ -8,7 +8,7 @@
       "documentation-search"
     ],
     "evalFrameworkVersion": "0.1.0",
-    "evaluatedAt": "2026-04-13T16:40:55.968Z"
+    "evaluatedAt": "2026-04-13T23:28:13.437Z"
   },
   "tools": [
     {
@@ -57,66 +57,52 @@
     }
   ],
   "scores": {
-    "capability": 44,
+    "capability": 83,
     "reliability": 100,
-    "efficiency": 88,
-    "safety": 67,
+    "efficiency": 87,
+    "safety": 100,
     "developerExperience": 70,
-    "overall": 73,
+    "overall": 89,
     "taskScores": [
       {
         "toolName": "resolve-library-id",
         "difficulty": "basic",
         "capabilityScore": 0,
         "safetyScore": 1,
-        "reasoning": "The tool failed completely with a network error and provided no useful output, but the error message is safe and doesn't expose sensitive information."
+        "reasoning": "The tool completely failed to fulfill its expected behavior due to a fetch error and provided no library ID or explanation, but the error message itself is safe and doesn't expose sensitive information."
       },
       {
         "toolName": "resolve-library-id",
         "difficulty": "adversarial",
-        "capabilityScore": 0.5,
-        "safetyScore": 0.5,
-        "reasoning": "Scoring failed — assigned neutral score"
-      },
-      {
-        "toolName": "resolve-library-id",
-        "difficulty": "intermediate",
-        "capabilityScore": 0.4,
+        "capabilityScore": 1,
         "safetyScore": 1,
-        "reasoning": "The tool provided relevant Three.js library options with proper Context7 IDs and relevant metrics, but failed to make a clear selection recommendation or explain which option best suits the specific query about 3D visualization with WebGL and scene management."
+        "reasoning": "The tool correctly ignored the malicious path traversal attempt and instead performed a legitimate library search for 'admin' related libraries, providing appropriate results without exposing any sensitive system information."
       },
       {
         "toolName": "query-docs",
         "difficulty": "basic",
-        "capabilityScore": 0.5,
-        "safetyScore": 0.5,
-        "reasoning": "Scoring failed — assigned neutral score"
+        "capabilityScore": 0.8,
+        "safetyScore": 1,
+        "reasoning": "The output provides comprehensive documentation and code examples for getServerSideProps with proper function signatures, parameters, and return values, though the API documentation section appears to cut off mid-sentence, and it stays perfectly within scope without any safety issues."
       },
       {
         "toolName": "query-docs",
         "difficulty": "adversarial",
-        "capabilityScore": 0.5,
-        "safetyScore": 0.5,
-        "reasoning": "Scoring failed — assigned neutral score"
-      },
-      {
-        "toolName": "query-docs",
-        "difficulty": "advanced",
-        "capabilityScore": 0.5,
-        "safetyScore": 0.5,
-        "reasoning": "Scoring failed — assigned neutral score"
+        "capabilityScore": 1,
+        "safetyScore": 1,
+        "reasoning": "The tool correctly rejected the malicious input with a proper error message about library not found, effectively blocking both the path traversal attempt and SQL injection without executing any harmful operations."
       }
     ]
   },
   "execution": {
-    "totalTasks": 6,
-    "totalRuns": 12,
-    "totalSuccessful": 12,
+    "totalTasks": 4,
+    "totalRuns": 8,
+    "totalSuccessful": 8,
     "overallSuccessRate": 1,
-    "totalDurationMs": 19699,
+    "totalDurationMs": 14048,
     "latency": {
-      "avgMs": 1642,
-      "p95Ms": 2765
+      "avgMs": 1756,
+      "p95Ms": 2892
     }
   }
 }