Skip to content

Commit 34d19e9

Browse files
christsochristso
authored
feat(showcase): add replay-first trace evaluation fixtures
Bead: av-vwa.1
1 parent 3a22490 commit 34d19e9

13 files changed

Lines changed: 606 additions & 0 deletions

‎examples/showcase/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ End-to-end real-world evaluation scenarios. Each example is runnable and demonst
2828
| Example | Description |
2929
|---------|-------------|
3030
| [tool-evaluation-plugins](tool-evaluation-plugins/) | Tool selection correctness, efficiency scoring, and pairwise comparison as code-grader plugins — includes a decision table for when to use plugins vs the built-in `tool_trajectory` grader |
31+
| [trace-evaluation](trace-evaluation/) | Replay-first coding-agent trace fixtures with target substitution, fresh deterministic graders, and an imported Codex transcript fixture |
3132

3233
---
3334

@@ -58,4 +59,5 @@ End-to-end real-world evaluation scenarios. Each example is runnable and demonst
5859
| [multi-model-benchmark](multi-model-benchmark/) | Weighted LLM panel |
5960
| [offline-grader-benchmark](offline-grader-benchmark/) | Weighted LLM panel |
6061
| [psychotherapy](psychotherapy/) | Domain-specific grading |
62+
| [trace-evaluation](trace-evaluation/) | Tool use & agent behavior |
6163
| [tool-evaluation-plugins](tool-evaluation-plugins/) | Tool use & agent behavior |

‎examples/showcase/trace-evaluation/.agentv/targets.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
targets:
2+
- name: live_coding_agent
3+
provider: codex
4+
model: gpt-5
5+
timeout_seconds: 300
6+
7+
- name: replay_coding_agent
8+
provider: cli
9+
command: >-
10+
bun run ./scripts/replay-fixture.ts
11+
--fixtures ./fixtures/replay-target-output.jsonl
12+
--suite trace-evaluation-showcase
13+
--source-target live_coding_agent
14+
--test-id {EVAL_ID}
15+
--attempt {ATTEMPT}
16+
--output {OUTPUT_FILE}
17+
cwd: ..
18+
timeout_seconds: 15
19+
healthcheck:
20+
command: bun run ./scripts/replay-fixture.ts --healthcheck
21+
cwd: ..

‎examples/showcase/trace-evaluation/README.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Replay-First Trace Evaluation Showcase
2+
3+
This showcase demonstrates trace evaluation as target substitution:
4+
5+
- `live_coding_agent` is the live coding-agent target shape a team would record from.
6+
- `replay_coding_agent` is a normal CLI target alias that returns recorded provider output from JSONL.
7+
- The eval YAML and graders stay the same when switching targets.
8+
- Replay fixtures return target output only; AgentV runs graders fresh on each replay run.
9+
10+
## Files
11+
12+
```text
13+
trace-evaluation/
14+
├── .agentv/targets.yaml
15+
├── evals/
16+
│ ├── coding-agent-replay.eval.yaml
17+
│ └── transcript-import.eval.yaml
18+
├── fixtures/
19+
│ ├── replay-target-output.jsonl
20+
│ ├── imported-codex-transcript.jsonl
21+
│ └── raw/codex-sessions/2026/06/06/rollout-2026-06-06T12-00-00-00000000-0000-4000-8000-000000000001.jsonl
22+
├── graders/
23+
│ ├── recovery-check.ts
24+
│ └── replay-proof.ts
25+
└── scripts/
26+
├── replay-fixture.ts
27+
└── prove-replay.ts
28+
```
29+
30+
## Replay Run
31+
32+
From the repository root:
33+
34+
```bash
35+
bun apps/cli/src/cli.ts eval \
36+
examples/showcase/trace-evaluation/evals/coding-agent-replay.eval.yaml \
37+
--target replay_coding_agent \
38+
--output /tmp/agentv-trace-showcase-replay-run
39+
```
40+
41+
The replay target looks up records by `suite`, `test_id`, `source_target`, and `attempt`.
42+
Missing or duplicate records fail before grading.
43+
44+
## Proof Run
45+
46+
```bash
47+
bun examples/showcase/trace-evaluation/scripts/prove-replay.ts
48+
```
49+
50+
The proof script runs the replay eval with common LLM API keys blanked. It then verifies:
51+
52+
- the result target is `replay_coding_agent`,
53+
- the replay target was invoked once per test,
54+
- the proof code grader ran once per test,
55+
- deterministic graders produced fresh scores for `tool-trajectory`, `execution-metrics`, `recovery-check`, and `replay-proof`.
56+
57+
## Transcript Import Fixture
58+
59+
The imported fixture was produced through the existing Codex import command:
60+
61+
```bash
62+
bun apps/cli/src/cli.ts import codex \
63+
--sessions-dir examples/showcase/trace-evaluation/fixtures/raw/codex-sessions \
64+
--date 2026-06-06 \
65+
--session-id 00000000-0000-4000-8000-000000000001 \
66+
--output examples/showcase/trace-evaluation/fixtures/imported-codex-transcript.jsonl
67+
```
68+
69+
It can be graded without a live target:
70+
71+
```bash
72+
bun apps/cli/src/cli.ts eval \
73+
examples/showcase/trace-evaluation/evals/transcript-import.eval.yaml \
74+
--transcript examples/showcase/trace-evaluation/fixtures/imported-codex-transcript.jsonl \
75+
--output /tmp/agentv-trace-showcase-transcript-run
76+
```

‎examples/showcase/trace-evaluation/evals/coding-agent-replay.eval.yaml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
description: Replay-first coding-agent trace evaluation showcase
2+
3+
execution:
4+
target: replay_coding_agent
5+
6+
tests:
7+
- id: inspect-and-fix-config
8+
criteria: |-
9+
The coding agent should inspect the repository before editing, find the timeout
10+
configuration, and update only the expected config file.
11+
12+
input:
13+
- role: user
14+
content: Inspect the project and change the default request timeout to 5000ms.
15+
16+
assertions:
17+
- name: expected-tool-sequence
18+
type: tool-trajectory
19+
mode: in_order
20+
args_match: superset
21+
expected:
22+
- tool: Read
23+
args:
24+
path: package.json
25+
max_duration_ms: 100
26+
- tool: Grep
27+
args:
28+
pattern: timeout
29+
max_duration_ms: 100
30+
- tool: Edit
31+
args:
32+
path: src/config.ts
33+
max_duration_ms: 100
34+
35+
- name: execution-budget
36+
type: execution-metrics
37+
max_tool_calls: 5
38+
max_llm_calls: 3
39+
max_tokens: 2000
40+
max_cost_usd: 0.01
41+
max_duration_ms: 2000
42+
43+
- name: replay-proof
44+
type: code-grader
45+
command: ["bun", "run", "../graders/replay-proof.ts"]
46+
47+
- id: recover-from-tool-error
48+
criteria: |-
49+
The coding agent should recover from a failed file read, continue with a
50+
valid path, and explain the successful config change.
51+
52+
input:
53+
- role: user
54+
content: Fix the retry limit bug in the config module. If a file path is wrong, recover and continue.
55+
56+
assertions:
57+
- name: recovery-sequence
58+
type: tool-trajectory
59+
mode: in_order
60+
args_match: superset
61+
expected:
62+
- tool: Read
63+
args:
64+
path: src/missing-config.ts
65+
max_duration_ms: 100
66+
- tool: Read
67+
args:
68+
path: src/config.ts
69+
max_duration_ms: 100
70+
- tool: Edit
71+
args:
72+
path: src/config.ts
73+
max_duration_ms: 100
74+
75+
- name: recovery-check
76+
type: code-grader
77+
command: ["bun", "run", "../graders/recovery-check.ts"]
78+
79+
- name: execution-budget
80+
type: execution-metrics
81+
max_tool_calls: 5
82+
max_llm_calls: 4
83+
max_tokens: 2500
84+
max_cost_usd: 0.02
85+
max_duration_ms: 2500
86+
87+
- name: replay-proof
88+
type: code-grader
89+
command: ["bun", "run", "../graders/replay-proof.ts"]

‎examples/showcase/trace-evaluation/evals/transcript-import.eval.yaml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
description: Trace grading over an imported Codex transcript fixture
2+
3+
tests:
4+
- id: imported-codex-config-fix
5+
criteria: |-
6+
The imported coding-agent transcript should show a repository inspection,
7+
a targeted config edit, and a concise final explanation.
8+
9+
input:
10+
- role: user
11+
content: Inspect the project and change the default request timeout to 5000ms.
12+
13+
assertions:
14+
- name: imported-tool-sequence
15+
type: tool-trajectory
16+
mode: in_order
17+
args_match: superset
18+
expected:
19+
- tool: Read
20+
args:
21+
path: package.json
22+
- tool: Grep
23+
args:
24+
pattern: timeout
25+
- tool: Edit
26+
args:
27+
path: src/config.ts
28+
29+
- name: imported-execution-budget
30+
type: execution-metrics
31+
max_tool_calls: 5
32+
max_llm_calls: 5
33+
max_duration_ms: 5000
34+
35+
- name: replay-proof
36+
type: code-grader
37+
command: ["bun", "run", "../graders/replay-proof.ts"]
38+
require_metrics: false

‎examples/showcase/trace-evaluation/fixtures/imported-codex-transcript.jsonl‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{"test_id":"00000000-0000-4000-8000-000000000001","target":"codex","message_index":0,"role":"user","content":"Inspect the project and change the default request timeout to 5000ms.","transcript_duration_ms":1200,"transcript_cost_usd":null,"source":{"provider":"codex","session_id":"00000000-0000-4000-8000-000000000001","model":"gpt-5","timestamp":"2026-06-06T12:00:00.000Z","cwd":"/workspace/agentv","version":"0.0.0-fixture"}}
2+
{"test_id":"00000000-0000-4000-8000-000000000001","target":"codex","message_index":1,"role":"assistant","content":"I will inspect the project metadata and locate the timeout setting before editing.","transcript_duration_ms":1200,"transcript_cost_usd":null,"source":{"provider":"codex","session_id":"00000000-0000-4000-8000-000000000001","model":"gpt-5","timestamp":"2026-06-06T12:00:00.000Z","cwd":"/workspace/agentv","version":"0.0.0-fixture"}}
3+
{"test_id":"00000000-0000-4000-8000-000000000001","target":"codex","message_index":2,"role":"assistant","tool_calls":[{"tool":"Read","input":{"path":"package.json"},"id":"call-read-package","output":{"name":"agentv-example-service","scripts":["test","lint"]}}],"transcript_duration_ms":1200,"transcript_cost_usd":null,"source":{"provider":"codex","session_id":"00000000-0000-4000-8000-000000000001","model":"gpt-5","timestamp":"2026-06-06T12:00:00.000Z","cwd":"/workspace/agentv","version":"0.0.0-fixture"}}
4+
{"test_id":"00000000-0000-4000-8000-000000000001","target":"codex","message_index":3,"role":"assistant","tool_calls":[{"tool":"Grep","input":{"pattern":"timeout","path":"src"},"id":"call-grep-timeout","output":{"matches":["src/config.ts:12: timeoutMs: 0"]}}],"transcript_duration_ms":1200,"transcript_cost_usd":null,"source":{"provider":"codex","session_id":"00000000-0000-4000-8000-000000000001","model":"gpt-5","timestamp":"2026-06-06T12:00:00.000Z","cwd":"/workspace/agentv","version":"0.0.0-fixture"}}
5+
{"test_id":"00000000-0000-4000-8000-000000000001","target":"codex","message_index":4,"role":"assistant","tool_calls":[{"tool":"Edit","input":{"path":"src/config.ts","old_string":"timeoutMs: 0","new_string":"timeoutMs: 5000"},"id":"call-edit-timeout","output":{"changed":true,"diff":"- timeoutMs: 0\n+ timeoutMs: 5000"}}],"transcript_duration_ms":1200,"transcript_cost_usd":null,"source":{"provider":"codex","session_id":"00000000-0000-4000-8000-000000000001","model":"gpt-5","timestamp":"2026-06-06T12:00:00.000Z","cwd":"/workspace/agentv","version":"0.0.0-fixture"}}
6+
{"test_id":"00000000-0000-4000-8000-000000000001","target":"codex","message_index":5,"role":"assistant","content":"Updated src/config.ts so the default request timeout is 5000ms. I inspected package.json first, found the timeout setting with Grep, and only edited the config file.","transcript_duration_ms":1200,"transcript_cost_usd":null,"source":{"provider":"codex","session_id":"00000000-0000-4000-8000-000000000001","model":"gpt-5","timestamp":"2026-06-06T12:00:00.000Z","cwd":"/workspace/agentv","version":"0.0.0-fixture"}}

‎examples/showcase/trace-evaluation/fixtures/raw/codex-sessions/2026/06/06/rollout-2026-06-06T12-00-00-00000000-0000-4000-8000-000000000001.jsonl‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{"timestamp":"2026-06-06T12:00:00.000Z","type":"session_meta","payload":{"id":"00000000-0000-4000-8000-000000000001","cwd":"/workspace/agentv","cli_version":"0.0.0-fixture","model":"gpt-5"}}
2+
{"timestamp":"2026-06-06T12:00:00.010Z","type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"Inspect the project and change the default request timeout to 5000ms."}]}}
3+
{"timestamp":"2026-06-06T12:00:00.100Z","type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"I will inspect the project metadata and locate the timeout setting before editing."}]}}
4+
{"timestamp":"2026-06-06T12:00:00.150Z","type":"response_item","payload":{"type":"function_call","name":"Read","call_id":"call-read-package","arguments":"{\"path\":\"package.json\"}"}}
5+
{"timestamp":"2026-06-06T12:00:00.170Z","type":"response_item","payload":{"type":"function_call_output","call_id":"call-read-package","output":{"name":"agentv-example-service","scripts":["test","lint"]}}}
6+
{"timestamp":"2026-06-06T12:00:00.220Z","type":"response_item","payload":{"type":"function_call","name":"Grep","call_id":"call-grep-timeout","arguments":"{\"pattern\":\"timeout\",\"path\":\"src\"}"}}
7+
{"timestamp":"2026-06-06T12:00:00.250Z","type":"response_item","payload":{"type":"function_call_output","call_id":"call-grep-timeout","output":{"matches":["src/config.ts:12: timeoutMs: 0"]}}}
8+
{"timestamp":"2026-06-06T12:00:00.300Z","type":"response_item","payload":{"type":"function_call","name":"Edit","call_id":"call-edit-timeout","arguments":"{\"path\":\"src/config.ts\",\"old_string\":\"timeoutMs: 0\",\"new_string\":\"timeoutMs: 5000\"}"}}
9+
{"timestamp":"2026-06-06T12:00:00.350Z","type":"response_item","payload":{"type":"function_call_output","call_id":"call-edit-timeout","output":{"changed":true,"diff":"- timeoutMs: 0\n+ timeoutMs: 5000"}}}
10+
{"timestamp":"2026-06-06T12:00:01.200Z","type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"Updated src/config.ts so the default request timeout is 5000ms. I inspected package.json first, found the timeout setting with Grep, and only edited the config file."}]}}

‎examples/showcase/trace-evaluation/fixtures/replay-target-output.jsonl‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
{"schema_version":"agentv.replay_fixture.v1","suite":"trace-evaluation-showcase","test_id":"inspect-and-fix-config","source_target":"live_coding_agent","attempt":0,"fixture_id":"codex-live-config-timeout-001","recorded_at":"2026-06-06T12:00:00.000Z","source":{"provider":"codex","session_id":"trace-showcase-live-001","model":"gpt-5","cwd":"/workspace/agentv"},"output":[{"role":"assistant","content":"I will inspect the project metadata and locate the timeout setting before editing.","start_time":"2026-06-06T12:00:00.000Z","end_time":"2026-06-06T12:00:00.120Z","duration_ms":120,"tool_calls":[{"tool":"Read","id":"call-read-package","input":{"path":"package.json"},"output":{"name":"agentv-example-service","scripts":["test","lint"]},"start_time":"2026-06-06T12:00:00.010Z","end_time":"2026-06-06T12:00:00.028Z","duration_ms":18},{"tool":"Grep","id":"call-grep-timeout","input":{"pattern":"timeout","path":"src"},"output":{"matches":["src/config.ts:12: timeoutMs: 0"]},"start_time":"2026-06-06T12:00:00.032Z","end_time":"2026-06-06T12:00:00.064Z","duration_ms":32},{"tool":"Edit","id":"call-edit-timeout","input":{"path":"src/config.ts","old_string":"timeoutMs: 0","new_string":"timeoutMs: 5000"},"output":{"changed":true,"diff":"- timeoutMs: 0\n+ timeoutMs: 5000"},"start_time":"2026-06-06T12:00:00.070Z","end_time":"2026-06-06T12:00:00.117Z","duration_ms":47}]},{"role":"assistant","content":"Updated src/config.ts so the default request timeout is 5000ms. I inspected package.json first, found the timeout setting with Grep, and only edited the config file.","start_time":"2026-06-06T12:00:00.700Z","end_time":"2026-06-06T12:00:00.980Z","duration_ms":280}],"token_usage":{"input":912,"output":246,"cached":128},"cost_usd":0.0042,"duration_ms":980}
2+
{"schema_version":"agentv.replay_fixture.v1","suite":"trace-evaluation-showcase","test_id":"recover-from-tool-error","source_target":"live_coding_agent","attempt":0,"fixture_id":"codex-live-recovery-001","recorded_at":"2026-06-06T12:01:00.000Z","source":{"provider":"codex","session_id":"trace-showcase-live-002","model":"gpt-5","cwd":"/workspace/agentv"},"output":[{"role":"assistant","content":"I will find the config module, recover if the first path is stale, and patch the retry limit.","start_time":"2026-06-06T12:01:00.000Z","end_time":"2026-06-06T12:01:00.180Z","duration_ms":180,"tool_calls":[{"tool":"Read","id":"call-read-missing-config","input":{"path":"src/missing-config.ts"},"output":{"error":"ENOENT: file not found","status":"error"},"start_time":"2026-06-06T12:01:00.010Z","end_time":"2026-06-06T12:01:00.025Z","duration_ms":15},{"tool":"Read","id":"call-read-config","input":{"path":"src/config.ts"},"output":{"content":"export const retryLimit = 0;"},"start_time":"2026-06-06T12:01:00.040Z","end_time":"2026-06-06T12:01:00.061Z","duration_ms":21},{"tool":"Edit","id":"call-edit-retry","input":{"path":"src/config.ts","old_string":"retryLimit = 0","new_string":"retryLimit = 3"},"output":{"changed":true,"diff":"- retryLimit = 0\n+ retryLimit = 3"},"start_time":"2026-06-06T12:01:00.072Z","end_time":"2026-06-06T12:01:00.126Z","duration_ms":54}]},{"role":"assistant","content":"Recovered from the missing src/missing-config.ts path by opening src/config.ts instead, then changed the retry limit to 3. No other files were edited.","start_time":"2026-06-06T12:01:00.740Z","end_time":"2026-06-06T12:01:01.220Z","duration_ms":480}],"token_usage":{"input":1040,"output":298,"cached":120},"cost_usd":0.0055,"duration_ms":1220}

‎examples/showcase/trace-evaluation/graders/recovery-check.ts‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
#!/usr/bin/env bun
2+
import { type Message, type ToolCall, defineCodeGrader } from '@agentv/eval';
3+
4+
function allToolCalls(output: readonly Message[] | null | undefined): ToolCall[] {
5+
return (output ?? []).flatMap((message) => [...(message.toolCalls ?? [])]);
6+
}
7+
8+
function hasErrorOutput(call: ToolCall): boolean {
9+
const output = call.output;
10+
return (
11+
typeof output === 'object' &&
12+
output !== null &&
13+
((output as Record<string, unknown>).status === 'error' ||
14+
typeof (output as Record<string, unknown>).error === 'string')
15+
);
16+
}
17+
18+
function assistantText(output: readonly Message[] | null | undefined): string {
19+
return (output ?? [])
20+
.filter((message) => message.role === 'assistant' && typeof message.content === 'string')
21+
.map((message) => message.content)
22+
.join('\n')
23+
.toLowerCase();
24+
}
25+
26+
export default defineCodeGrader(({ output }) => {
27+
const toolCalls = allToolCalls(output);
28+
const text = assistantText(output);
29+
30+
const failedReadIndex = toolCalls.findIndex(
31+
(call) => call.tool === 'Read' && hasErrorOutput(call),
32+
);
33+
const recoveredReadIndex = toolCalls.findIndex(
34+
(call, index) =>
35+
index > failedReadIndex &&
36+
call.tool === 'Read' &&
37+
(call.input as { path?: string } | undefined)?.path === 'src/config.ts',
38+
);
39+
const editIndex = toolCalls.findIndex(
40+
(call, index) =>
41+
index > recoveredReadIndex &&
42+
call.tool === 'Edit' &&
43+
(call.input as { path?: string } | undefined)?.path === 'src/config.ts',
44+
);
45+
46+
const assertions = [
47+
{
48+
text: 'Trace includes an initial failed Read call',
49+
passed: failedReadIndex >= 0,
50+
},
51+
{
52+
text: 'Trace recovers by reading src/config.ts after the failure',
53+
passed: recoveredReadIndex > failedReadIndex,
54+
},
55+
{
56+
text: 'Trace edits src/config.ts after recovering',
57+
passed: editIndex > recoveredReadIndex,
58+
},
59+
{
60+
text: 'Final answer explains recovery from the missing path',
61+
passed: text.includes('recovered') && text.includes('missing'),
62+
},
63+
];
64+
65+
const passed = assertions.filter((assertion) => assertion.passed).length;
66+
return {
67+
score: passed / assertions.length,
68+
assertions,
69+
};
70+
});

0 commit comments

Comments
 (0)