eval-assert-demo

Eval Assert Demo

Demonstrates script graders that can be run both as part of an eval suite and individually via agentv eval assert.

Graders

File	Purpose
.agentv/graders/keyword-check.ts	Checks answer contains expected keywords (Paris, France)
.agentv/graders/length-check.ts	Validates answer word count is between 5 and 50

Both graders use defineScriptGrader from @agentv/sdk.

Running the Full Eval

# From the repository root
bun agentv eval examples/features/eval-assert-demo/evals/suite.yaml

Running Assertions Individually

Run a single assertion without executing the full eval suite:

cd examples/features/eval-assert-demo

# Run keyword-check with inline args
bun agentv eval assert keyword-check \
  --agent-output "The capital of France is Paris." \
  --agent-input "What is the capital of France?"

# Run length-check
bun agentv eval assert length-check \
  --agent-output "The capital of France is Paris." \
  --agent-input "What is the capital of France?"

# Run from a JSON file
echo '{"output": "The capital of France is Paris.", "input": "What is the capital?"}' > result.json
bun agentv eval assert keyword-check --file result.json

Exit code is 0 if score >= 0.5 (pass), 1 otherwise (fail).

Inspecting Grading Criteria

bun agentv eval prompt eval --grading-brief \
  examples/features/eval-assert-demo/evals/suite.yaml \
  --test-id capital-of-france

Output:

Input: "What is the capital of France? Answer in one concise sentence."
Expected: "The capital of France is Paris."
Criteria:
  - Output contains 'Paris'
  - [script-grader] keyword-check: Checks that the answer mentions Paris and France
  - [script-grader] length-check: Ensures answer is between 5 and 50 words

How It Works

When running the eval, the transpiler emits natural-language instructions for each script grader:

Run `agentv eval assert keyword-check --agent-output <text> --agent-input <text>` and check the result.
This grader: Checks that the answer mentions Paris and France.
The command returns JSON: {"score": 0-1, "assertions": [{"text": "...", "passed": true|false}]}.
A score >= 0.5 means pass (exit 0); below 0.5 means fail (exit 1).

This allows external grading agents to execute script graders directly without understanding their internal implementation.