Skip to content

Commit 2762691

Browse files
iamzifeiclaude
andcommitted
docs: update Dev.to blog with v0.2.0 fixes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 5a6b085 commit 2762691

1 file changed

Lines changed: 4 additions & 4 deletions

File tree

docs/blog/devto-mcp-benchmark.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -112,17 +112,17 @@ Output looks like this:
112112

113113
## Caveats
114114

115-
- **LLM non-determinism**: Scores vary ±5 points between runs because both task generation and judging use Claude. Deterministic task sets are coming in v0.2.
115+
- ~~**LLM non-determinism**~~: **Fixed in v0.2.0** — tasks are now cached after first generation. Subsequent runs reuse the same tasks for reproducible scores. Use `--regenerate-tasks` to force new tasks.
116116
- **Auto-generated tasks**: The framework generates test tasks from tool schemas. For tools that need real-world context (file systems with actual files, databases with actual data), reliability scores will be lower than real-world usage.
117-
- **DX score is a placeholder**: Developer Experience is scored at a flat 70 for now. Proper DX evaluation (docs quality, error message helpfulness) is coming.
118-
- **Single model judge**: Using Claude to judge Claude-generated tasks has inherent bias. Multi-model judging is on the roadmap.
117+
- ~~**DX score is a placeholder**~~: **Fixed in v0.2.0** — DX now scores schema quality (typed properties, descriptions), documentation (meaningful tool descriptions), and error messages (informative vs empty) automatically. No more flat 70.
118+
- **Single model judge**: Using Claude to judge Claude-generated tasks has inherent bias. v0.2.0 adds `eval.judge.model` config to switch models. Multi-provider judging is on the roadmap.
119119

120120
## What's next
121121

122122
- **A2A protocol support** — evaluate Google's Agent-to-Agent servers
123-
- **Deterministic task sets** — curated test suites per category
124123
- **Web dashboard** — browse rankings at eval.agenthunter.io
125124
- **Continuous monitoring** — track score changes over time
125+
- **Multi-provider judging** — use GPT-4o, Gemini, or multiple models as judges
126126

127127
The framework is fully open source: **[github.com/OrrisTech/agent-eval](https://github.com/OrrisTech/agent-eval)**
128128

0 commit comments

Comments
 (0)