@@ -95,6 +95,9 @@ Fast, stateless, composable operations for mid-task self-evaluation:
9595Stateful, longer-running operations for eval-driven development. Accepts inline
9696trajectory objects (not file paths). Storage is in-memory per session.
9797
98+ > ** Note:** The suite MCP tools use a mock evaluator that always returns score=0.85.
99+ > For real evaluation, use the library API or CLI directly.
100+
98101| Tool | Input | Output | Use Case |
99102| ------| -------| --------| ----------|
100103| ` eval.suite.run ` | ` { trajectories, config? } ` | ` { run_id, status, total_trajectories, completed, failed, duration_ms } ` | Execute full evaluation suite |
@@ -232,14 +235,14 @@ The `GoldenCurator` class provides a structured curation workflow:
232235import { createCurator , quickCreateGolden } from ' @reaatech/agent-eval-harness' ;
233236
234237// Full curation workflow (identify → annotate → validate → publish)
235- const curator = createCurator (' my_suite ' );
236- curator .start (trajectory );
237- curator .annotateTurn (0 , ' Polite greeting' , true );
238+ const curator = createCurator (trajectory );
239+ curator .start ({ scenario: ' password-reset ' } );
240+ curator .annotateTurn ({ turnId: 1 , expected: true , qualityNotes: ' Polite greeting' } );
238241curator .runQualityChecks ();
239242const golden = curator .publish ();
240243
241244// Quick creation for simple scenarios
242- const golden = quickCreateGolden (trajectory , { scenario: ' password-reset' } );
245+ const golden = quickCreateGolden (trajectory , ' password-reset' , [ ' auth ' ] );
243246```
244247
245248---
@@ -347,14 +350,14 @@ cost:
347350 input : 30.00
348351 output : 60.00
349352 gpt-4-mini :
350- input : 0.15
351- output : 0.60
353+ input : 3.00
354+ output : 10.00
352355 gemini-pro :
353356 input : 2.50
354357 output : 7.50
355358 gemini-flash :
356- input : 0.50
357- output : 1.50
359+ input : 0.075
360+ output : 0.30
358361
359362 # Budget settings
360363 budgets :
@@ -378,9 +381,9 @@ Three named presets are available for quick setup:
378381
379382| Preset | Per Task | Per Trajectory | Daily |
380383|--------|----------|----------------|-------|
381- | ` strict` | $0.02 | $0.50 | $50 .00 |
382- | `moderate` | $0.05 | $1.00 | $100 .00 |
383- | `lenient` | $0.10 | $2 .00 | $250 .00 |
384+ | ` strict` | $0.01 | $0.50 | $10 .00 |
385+ | `moderate` | $0.05 | $1.00 | $50 .00 |
386+ | `lenient` | $0.10 | $5 .00 | $100 .00 |
384387
385388# ## Cost Breakdown
386389
@@ -660,9 +663,9 @@ Three presets provide ready-made gate configurations:
660663
661664| Preset | Overall Quality | Cost | Latency P99 | Tool Correctness | Faithfulness |
662665|--------|----------------|------|-------------|------------------|--------------|
663- | **standard** | >= 0.80 | <= $0.05 | <= 5000ms | >= 0.95 | >= 0.85 |
664- | **strict** | >= 0.90 | <= $0.03 | <= 3000ms | >= 0.98 | >= 0.90 |
665- | **lenient** | >= 0.70 | <= $0.10 | <= 10000ms | >= 0.85 | >= 0.75 |
666+ | **standard** | >= 0.80 | <= $0.05 | <= 5000ms | >= 0.90 | >= 0.80 |
667+ | **strict** | >= 0.90 | <= $0.02 | <= 2000ms | >= 0.95 | >= 0.90 |
668+ | **lenient** | >= 0.60 | <= $0.10 | <= 10000ms | >= 0.70 | >= 0.60 |
666669
667670### Programmatic Gate Construction
668671
@@ -709,8 +712,7 @@ npx agent-eval-harness eval trajectories/*.jsonl \
709712npx agent-eval-harness judge faithfulness \
710713 --context " The user's account is associated with email john@example.com" \
711714 --response " I've sent the password reset to john@example.com" \
712- --model claude-opus \
713- --calibrated
715+ --model claude-opus
714716
715717# Compare two runs (exits 1 on regressions)
716718npx agent-eval-harness compare results/baseline.json results/candidate.json \
0 commit comments