Skip to content

Commit ab28008

Browse files
iamzifeiclaude
andcommitted
docs: add content generation rules + update blog with verified data
CLAUDE.md: Added "Content Generation Rules" section — always check system date, re-read latest results files, include generation date, use exact numbers from data. Blog post: Rewrote with per-task timing data from actual results, added evaluation date (April 15, 2026), linked to eval.agenthunter.io. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent cad7ef1 commit ab28008

2 files changed

Lines changed: 29 additions & 20 deletions

File tree

CLAUDE.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# AgentHunter Eval
22

3-
The credit rating agency for AI agents — open-source evaluation framework + discovery platform.
3+
The credit rating agency for AI agents — open-source evaluation platform with two layers: Task Eval (agent task completion) + Tool Eval (MCP server quality).
44

55
## Quick Reference
66

@@ -63,6 +63,13 @@ agent-eval/
6363
- ESM-only (`"type": "module"` in package.json)
6464
- Use `node:` prefix for Node.js built-in imports
6565

66+
## Content Generation Rules
67+
68+
- **CRITICAL**: When generating any content (blog posts, reports, documentation), always check the current system date first and ensure all data, statistics, and references are up-to-date as of that date
69+
- Never use stale data from earlier in the conversation — always re-read the latest results/ files before generating content
70+
- Include the generation date in all published content
71+
- Reference specific evaluation data with exact numbers from the latest results
72+
6673
## Key Design Decisions
6774

6875
1. **Protocol adapters are pluggable** — each protocol (MCP, A2A, REST) implements a common `ProtocolAdapter` interface

docs/blog/devto-3-agents-10-tasks.md

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ canonical_url: https://github.com/OrrisTech/agent-eval
77
cover_image:
88
---
99

10+
*Evaluated on April 15, 2026 using [AgentHunter Eval](https://eval.agenthunter.io) v0.3.1*
11+
1012
Which Claude model should you use for your agent? I tested all three on the same 10 tasks to find out.
1113

1214
## Setup
@@ -32,23 +34,23 @@ Sonnet and Opus both scored perfect 10/10. Haiku missed one task but was **2x fa
3234

3335
### Coding (5 tasks)
3436

35-
| Task | Sonnet | Haiku | Opus |
36-
|------|--------|-------|------|
37-
| Create a CLI tool | PASS | PASS | PASS |
38-
| Fix a sorting bug | PASS | PASS | PASS |
39-
| Analyze CSV data | PASS | **FAIL** | PASS |
40-
| Write unit tests | PASS | PASS | PASS |
41-
| Refactor repetitive code | PASS | PASS | PASS |
37+
| Task | Sonnet 4 | Haiku 4.5 | Opus 4 |
38+
|------|----------|-----------|--------|
39+
| Create a CLI tool | PASS (5.0s) | PASS (3.4s) | PASS (6.6s) |
40+
| Fix a sorting bug | PASS (3.5s) | PASS (2.6s) | PASS (3.7s) |
41+
| Analyze CSV data | PASS (5.2s) | **FAIL** (3.4s) | PASS (6.5s) |
42+
| Write unit tests | PASS (9.9s) | PASS (5.5s) | PASS (12.8s) |
43+
| Refactor repetitive code | PASS (4.2s) | PASS (2.6s) | PASS (6.5s) |
4244

4345
### Writing & Docs (5 tasks)
4446

45-
| Task | Sonnet | Haiku | Opus |
46-
|------|--------|-------|------|
47-
| Write a professional email | PASS | PASS | PASS |
48-
| Summarize a technical doc | PASS | PASS | PASS |
49-
| Create a backup shell script | PASS | PASS | PASS |
50-
| Convert JSON to CSV | PASS | PASS | PASS |
51-
| Write a project README | PASS | PASS | PASS |
47+
| Task | Sonnet 4 | Haiku 4.5 | Opus 4 |
48+
|------|----------|-----------|--------|
49+
| Write a professional email | PASS (13.3s) | PASS (4.5s) | PASS (11.6s) |
50+
| Summarize a technical doc | PASS (8.8s) | PASS (4.0s) | PASS (9.2s) |
51+
| Create a backup shell script | PASS (5.6s) | PASS (3.3s) | PASS (6.3s) |
52+
| Convert JSON to CSV | PASS (11.6s) | PASS (5.8s) | PASS (10.6s) |
53+
| Write a project README | PASS (20.0s) | PASS (8.7s) | PASS (20.1s) |
5254

5355
## Key Findings
5456

@@ -75,10 +77,7 @@ At 4.4s average (2x faster than Sonnet), Haiku is the clear choice when speed ma
7577
## How to replicate
7678

7779
```bash
78-
# Install
79-
npm install -g @agenthunter/eval
80-
81-
# Run a task evaluation
80+
# Create a task definition
8281
cat > task.yaml << 'EOF'
8382
task:
8483
name: "My task"
@@ -95,6 +94,7 @@ eval:
9594
runs: 1
9695
EOF
9796

97+
# Run evaluation
9898
ANTHROPIC_API_KEY=your-key npx @agenthunter/eval task
9999
```
100100

@@ -109,6 +109,8 @@ ANTHROPIC_API_KEY=your-key npx @agenthunter/eval task
109109

110110
All evaluation data is open: [github.com/OrrisTech/agent-eval/results](https://github.com/OrrisTech/agent-eval/tree/main/results)
111111

112+
Full interactive results: [eval.agenthunter.io](https://eval.agenthunter.io)
113+
112114
---
113115

114-
*Built with [AgentHunter Eval](https://agent-eval-cyan.vercel.app) — the open-source AI agent evaluation platform. Try it: `npx @agenthunter/eval task`*
116+
*Built with [AgentHunter Eval](https://eval.agenthunter.io) — the open-source AI agent evaluation platform. Try it: `npx @agenthunter/eval task`*

0 commit comments

Comments
 (0)