Skip to content

Commit adcb6cf

Browse files
christsochristso
authored
docs(evals): document benchmark provenance patterns (#1304)
1 parent 20bd753 commit adcb6cf

3 files changed

Lines changed: 274 additions & 1 deletion

File tree

‎apps/web/src/content/docs/docs/evaluation/eval-cases.mdx‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,8 @@ tests:
159159

160160
The `metadata` field is included in the stdin JSON passed to lifecycle commands as `case_metadata`.
161161
Operational checkout state belongs under `workspace.repos[].checkout.base_commit`; `metadata.base_commit` is informational only.
162+
For benchmark task packs with source pins, patches, generated rows, and
163+
supporting files, see [Benchmark Provenance](/docs/guides/benchmark-provenance/).
162164

163165
## Per-Test Assertions
164166

@@ -389,7 +391,7 @@ tests:
389391

390392
## Metadata
391393

392-
Pass additional context to graders via the `metadata` field:
394+
Pass additional context through the `metadata` field:
393395

394396
```yaml
395397
tests:
@@ -400,3 +402,8 @@ tests:
400402
difficulty: medium
401403
input: Write a function to sort a list
402404
```
405+
406+
`metadata` is passed to workspace lifecycle hooks as `case_metadata`, preserved
407+
in result records, and available to in-process custom assertions. AgentV does
408+
not interpret arbitrary metadata keys itself; use `workspace`, `execution`,
409+
`input`, `expected_output`, and `assertions` for operational behavior.

‎apps/web/src/content/docs/docs/evaluation/eval-files.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,8 @@ input: Fix the null check bug in parser.ts
217217
- **Suite-level config applies:** Suite-level `assertions`, `input`, `workspace`, and `execution` still apply to directory-discovered cases
218218

219219
This pattern is useful for benchmarks with many cases, where each case benefits from its own directory for workspace templates, supporting files, or documentation.
220+
For guidance on keeping provenance metadata, patches, oracle files, and generated
221+
dataset rows out of oversized inline YAML, see [Benchmark Provenance](/docs/guides/benchmark-provenance/).
220222

221223
## Environment Variable Interpolation
222224

‎apps/web/src/content/docs/docs/guides/benchmark-provenance.mdx‎

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
---
2+
title: Benchmark Provenance
3+
description: Patterns for source pins, task artifacts, hooks, and generated benchmark metadata.
4+
sidebar:
5+
order: 5
6+
---
7+
8+
Benchmark suites usually need more than a prompt and a score. They carry source
9+
pins, task patches, generated dataset rows, oracle data, setup scripts, and
10+
verification commands. AgentV represents that with existing primitives:
11+
12+
- Put runtime behavior in `workspace`, `execution`, `input`, `expected_output`,
13+
and `assertions`.
14+
- Put provenance and classification in per-case `metadata`.
15+
- Put bulky per-case artifacts in case directories and supporting files.
16+
17+
These are documentation patterns, not special runtime schema keys. AgentV does
18+
not interpret keys such as `source_commit`, `test_patch`, or `question_type`
19+
unless your hook or custom assertion reads them.
20+
21+
## Operational vs Informational Fields
22+
23+
Use this split when deciding where a benchmark key belongs:
24+
25+
| Field area | Operational? | What AgentV does |
26+
|------------|--------------|------------------|
27+
| `workspace.repos[]` | Yes | Clones or copies repositories and checks out the configured refs. |
28+
| `workspace.template` | Yes | Copies a workspace template into the run workspace. |
29+
| `workspace.hooks` | Yes | Runs lifecycle commands with workspace and case context on stdin. |
30+
| `workspace.isolation`, `workspace.mode`, `workspace.path` | Yes | Controls workspace reuse and materialization. |
31+
| `execution` | Yes | Selects targets, thresholds, dependencies, and default grader behavior. |
32+
| `input`, `input_files`, `expected_output` | Yes | Builds the target prompt and passive reference answer. |
33+
| `assertions` | Yes | Runs deterministic, LLM, composite, or code graders. |
34+
| Top-level `name`, `version`, `tags`, `license`, `requires` | Informational | Identifies and categorizes the suite. |
35+
| `tests[].metadata` | Informational to AgentV | Passes arbitrary case data through to results and hook stdin; in-process custom assertions can also read it. |
36+
37+
`metadata` can still become operational inside your own hook scripts. For
38+
example, a `before_each` hook can read `case_metadata.test_patch` and apply that
39+
patch before the agent starts. The distinction is that AgentV itself only passes
40+
the metadata along; the script owns the behavior.
41+
42+
## Hook Payloads
43+
44+
Lifecycle hooks receive JSON on stdin. Case-scoped hooks such as per-test
45+
`before_all`, `before_each`, and `after_each` receive the current test's
46+
metadata as `case_metadata`:
47+
48+
```json
49+
{
50+
"workspace_path": "/home/user/.agentv/workspaces/run-123/case-01",
51+
"test_id": "case-01",
52+
"eval_run_id": "run-123",
53+
"case_input": "Fix the bug",
54+
"case_metadata": {
55+
"source_commit": "4f3e2d1",
56+
"test_patch": "cases/case-01/test.patch"
57+
}
58+
}
59+
```
60+
61+
Suite-level `before_all` hooks run once for the workspace, before any one test is
62+
selected, so they should do suite setup only. Use `before_each` when setup depends
63+
on per-case metadata such as a patch path, source row, or selected test list.
64+
65+
## Task Artifact Anatomy
66+
67+
Benchmark task packs map cleanly onto AgentV fields:
68+
69+
| Task artifact | AgentV pattern |
70+
|---------------|----------------|
71+
| Prompt or instruction | `input`, usually with `type: file` blocks for long prompts |
72+
| Source checkout | `workspace.repos[].source` and `workspace.repos[].checkout` |
73+
| Per-case setup | `workspace.hooks.before_each` reading `case_metadata` |
74+
| Gold answer | `expected_output` when the answer is passive reference data |
75+
| Active verification | `assertions`, especially `code-grader` for commands or artifact checks |
76+
| Provenance | `tests[].metadata` with source pins, generator rows, and curation labels |
77+
| Bulky task files | `tests: ./cases/` with per-case directories and supporting files |
78+
79+
This mirrors the common task shape used by filesystem-native benchmark harnesses:
80+
Margin keeps each task's prompt, case metadata, tests, environment, and optional
81+
oracle in a case directory; Terminal-Bench and Harbor keep task instructions,
82+
container setup, run-test scripts, and result artifacts as separate files. In
83+
AgentV, keep the same separation but bind it with eval YAML instead of adding a
84+
large benchmark-specific schema.
85+
86+
## SWE-Style Case
87+
88+
A SWE-style benchmark usually needs a source repo, a commit pin, a patch that
89+
adds or selects tests, and a list of failing tests that should pass after the
90+
agent's fix. Keep the checkout operational under `workspace.repos`; keep the
91+
benchmark provenance and per-case test selectors in `metadata`.
92+
93+
```yaml
94+
name: swe-style-regression
95+
description: Regression tasks against pinned source commits.
96+
97+
workspace:
98+
isolation: per_test
99+
repos:
100+
- path: ./repo
101+
source:
102+
type: git
103+
url: https://github.com/example/widget.git
104+
checkout:
105+
ref: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123
106+
clone:
107+
depth: 1
108+
hooks:
109+
before_each:
110+
command: ["python", "./scripts/apply-test-patch.py"]
111+
timeout_ms: 120000
112+
after_each:
113+
reset: strict
114+
115+
assertions:
116+
- name: focused-tests
117+
type: code-grader
118+
command: ["python", "./graders/run-focused-tests.py"]
119+
required: true
120+
121+
tests:
122+
- id: widget-1234
123+
criteria: Fix the widget parser regression without breaking existing behavior.
124+
input: |
125+
Work in repo/. Fix the parser regression described by the failing tests.
126+
Do not change unrelated public APIs.
127+
metadata:
128+
repo_url: https://github.com/example/widget.git
129+
source_commit: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123
130+
test_patch: cases/widget-1234/test.patch
131+
fail_to_pass_tests:
132+
- tests/parser.test.ts::handles-empty-widget
133+
- tests/parser.test.ts::preserves-widget-id
134+
```
135+
136+
In this example, `workspace.repos[].checkout.ref` is the actual checkout. The
137+
matching `metadata.source_commit` is audit data that gets recorded with the case
138+
and is available to scripts. `apply-test-patch.py` can read
139+
`case_metadata.test_patch` and `case_metadata.fail_to_pass_tests`, then apply
140+
the patch and write the selected test list into the workspace. The code grader
141+
can read that workspace file through its `workspace_path` payload.
142+
143+
## Finance-Style Generated Dataset
144+
145+
Generated datasets often need stable row provenance more than workspace setup.
146+
Keep the generated row identity in metadata, use `expected_output` for the gold
147+
answer, and score with rubrics or an LLM/code grader.
148+
149+
```yaml
150+
name: finance-research-generated
151+
description: Generated finance research cases with row-level provenance.
152+
153+
assertions:
154+
- name: answer-quality
155+
type: llm-grader
156+
prompt: ./graders/finance-answer.md
157+
required: true
158+
159+
tests:
160+
- id: finance-agent-row-0042
161+
criteria: Answer the finance question with the correct conclusion and evidence.
162+
input: |
163+
Research the company filing and answer:
164+
What drove the year-over-year change in gross margin?
165+
expected_output:
166+
- role: assistant
167+
content: |
168+
Gross margin improved because product mix shifted toward higher-margin
169+
software revenue while fulfillment costs declined.
170+
metadata:
171+
source_repo: https://github.com/example/finance-research-dataset.git
172+
source_commit: 05b8b2e9f071e8d0a6f1c2b3d4e5f60718293abc
173+
source_file: data/generated/finance_agent.csv
174+
source_row: 42
175+
question_type: margin_analysis
176+
```
177+
178+
Here, `source_repo`, `source_commit`, `source_file`, `source_row`, and
179+
`question_type` are informational metadata. They support audits, slices, and
180+
regeneration checks. If a hook or grader needs the source file at runtime, clone
181+
it through `workspace.repos` or make the generator output available as a normal
182+
fixture file.
183+
184+
## When to Split Into Case Directories
185+
186+
Inline YAML is fine when a case has a short prompt, a short expected answer, and
187+
a few metadata fields. Move away from inline YAML when the benchmark starts
188+
accumulating task-local artifacts:
189+
190+
- The case has patches, hidden tests, oracle JSON, screenshots, reports, or
191+
fixture files.
192+
- The prompt or expected output is long enough that YAML diffs become hard to
193+
review.
194+
- Each task needs a different workspace template or setup files.
195+
- A generator emits many rows and reviewers need to inspect individual cases.
196+
- Hook and grader scripts need stable file paths for per-case resources.
197+
198+
Use an external YAML or JSONL file for many simple generated rows:
199+
200+
```yaml
201+
name: generated-finance
202+
tests: ./cases.jsonl
203+
```
204+
205+
Use case directories when each case needs supporting files:
206+
207+
```text
208+
swe-benchmark/
209+
EVAL.yaml
210+
cases/
211+
widget-1234/
212+
case.yaml
213+
prompt.md
214+
test.patch
215+
oracle.json
216+
workspace/
217+
README.md
218+
```
219+
220+
```yaml
221+
# EVAL.yaml
222+
name: swe-benchmark
223+
workspace:
224+
repos:
225+
- path: ./repo
226+
source: { type: git, url: https://github.com/example/widget.git }
227+
checkout: { ref: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123 }
228+
tests: ./cases/
229+
```
230+
231+
```yaml
232+
# cases/widget-1234/case.yaml
233+
criteria: Fix the widget parser regression.
234+
input:
235+
- role: user
236+
content:
237+
- type: file
238+
value: cases/widget-1234/prompt.md
239+
metadata:
240+
repo_url: https://github.com/example/widget.git
241+
source_commit: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123
242+
test_patch: cases/widget-1234/test.patch
243+
oracle_file: cases/widget-1234/oracle.json
244+
```
245+
246+
When `tests` points to a directory, AgentV discovers each immediate
247+
subdirectory's `case.yaml`, uses the directory name as `id` if no `id` is set,
248+
and automatically uses a `workspace/` subdirectory as that case's
249+
`workspace.template`. File blocks still use the normal eval-file search roots,
250+
so include the case directory in paths such as `cases/widget-1234/prompt.md`.
251+
Metadata paths are not resolved by AgentV; resolve them in your hook or grader
252+
script.
253+
254+
## Authoring Rules
255+
256+
- Do not add benchmark-specific fields when `metadata` plus hooks or custom
257+
assertions can express the need.
258+
- Do not duplicate operational checkout state only in metadata. Put the real
259+
checkout under `workspace.repos`.
260+
- Keep `metadata` snake_case because it crosses process and result boundaries.
261+
- Prefer `expected_output` for passive gold answers and `code-grader` for active
262+
commands, file checks, or generated artifact validation.
263+
- Prefer case directories over long inline YAML once task artifacts become part
264+
of the benchmark contract.

0 commit comments

Comments
 (0)