Skip to content

Commit a2185b3

Browse files
committed
docs: fix inaccurate pricing, presets, and API examples in AGENTS.md
Corrected 5 categories of errors against source implementation: - Model pricing: gpt-4-mini 0.15/0.60 -> 3.00/10.00, gemini-flash 0.50/1.50 -> 0.075/0.30 - Budget presets: strict per_task 0.02->0.01 daily 50->10, lenient per_traj 2->5 daily 250->100 - Gate presets: standard tool 0.95->0.90, lenient quality 0.70->0.60 tool 0.85->0.70 - Golden curator API: createCurator takes Trajectory not string, annotateTurn takes TurnAnnotation object - Added note that MCP suite tools use mock evaluator (score=0.85)
1 parent fab3909 commit a2185b3

1 file changed

Lines changed: 18 additions & 16 deletions

File tree

AGENTS.md

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,9 @@ Fast, stateless, composable operations for mid-task self-evaluation:
9595
Stateful, longer-running operations for eval-driven development. Accepts inline
9696
trajectory objects (not file paths). Storage is in-memory per session.
9797

98+
> **Note:** The suite MCP tools use a mock evaluator that always returns score=0.85.
99+
> For real evaluation, use the library API or CLI directly.
100+
98101
| Tool | Input | Output | Use Case |
99102
|------|-------|--------|----------|
100103
| `eval.suite.run` | `{ trajectories, config? }` | `{ run_id, status, total_trajectories, completed, failed, duration_ms }` | Execute full evaluation suite |
@@ -232,14 +235,14 @@ The `GoldenCurator` class provides a structured curation workflow:
232235
import { createCurator, quickCreateGolden } from '@reaatech/agent-eval-harness';
233236

234237
// Full curation workflow (identify → annotate → validate → publish)
235-
const curator = createCurator('my_suite');
236-
curator.start(trajectory);
237-
curator.annotateTurn(0, 'Polite greeting', true);
238+
const curator = createCurator(trajectory);
239+
curator.start({ scenario: 'password-reset' });
240+
curator.annotateTurn({ turnId: 1, expected: true, qualityNotes: 'Polite greeting' });
238241
curator.runQualityChecks();
239242
const golden = curator.publish();
240243

241244
// Quick creation for simple scenarios
242-
const golden = quickCreateGolden(trajectory, { scenario: 'password-reset' });
245+
const golden = quickCreateGolden(trajectory, 'password-reset', ['auth']);
243246
```
244247

245248
---
@@ -347,14 +350,14 @@ cost:
347350
input: 30.00
348351
output: 60.00
349352
gpt-4-mini:
350-
input: 0.15
351-
output: 0.60
353+
input: 3.00
354+
output: 10.00
352355
gemini-pro:
353356
input: 2.50
354357
output: 7.50
355358
gemini-flash:
356-
input: 0.50
357-
output: 1.50
359+
input: 0.075
360+
output: 0.30
358361

359362
# Budget settings
360363
budgets:
@@ -378,9 +381,9 @@ Three named presets are available for quick setup:
378381
379382
| Preset | Per Task | Per Trajectory | Daily |
380383
|--------|----------|----------------|-------|
381-
| `strict` | $0.02 | $0.50 | $50.00 |
382-
| `moderate` | $0.05 | $1.00 | $100.00 |
383-
| `lenient` | $0.10 | $2.00 | $250.00 |
384+
| `strict` | $0.01 | $0.50 | $10.00 |
385+
| `moderate` | $0.05 | $1.00 | $50.00 |
386+
| `lenient` | $0.10 | $5.00 | $100.00 |
384387

385388
### Cost Breakdown
386389

@@ -660,9 +663,9 @@ Three presets provide ready-made gate configurations:
660663
661664
| Preset | Overall Quality | Cost | Latency P99 | Tool Correctness | Faithfulness |
662665
|--------|----------------|------|-------------|------------------|--------------|
663-
| **standard** | >= 0.80 | <= $0.05 | <= 5000ms | >= 0.95 | >= 0.85 |
664-
| **strict** | >= 0.90 | <= $0.03 | <= 3000ms | >= 0.98 | >= 0.90 |
665-
| **lenient** | >= 0.70 | <= $0.10 | <= 10000ms | >= 0.85 | >= 0.75 |
666+
| **standard** | >= 0.80 | <= $0.05 | <= 5000ms | >= 0.90 | >= 0.80 |
667+
| **strict** | >= 0.90 | <= $0.02 | <= 2000ms | >= 0.95 | >= 0.90 |
668+
| **lenient** | >= 0.60 | <= $0.10 | <= 10000ms | >= 0.70 | >= 0.60 |
666669
667670
### Programmatic Gate Construction
668671
@@ -709,8 +712,7 @@ npx agent-eval-harness eval trajectories/*.jsonl \
709712
npx agent-eval-harness judge faithfulness \
710713
--context "The user's account is associated with email john@example.com" \
711714
--response "I've sent the password reset to john@example.com" \
712-
--model claude-opus \
713-
--calibrated
715+
--model claude-opus
714716

715717
# Compare two runs (exits 1 on regressions)
716718
npx agent-eval-harness compare results/baseline.json results/candidate.json \

0 commit comments

Comments
 (0)