Skip to content

Commit 0d9c676

Browse files
authored
docs(migration): align skill and trajectory assertion docs (#1687)
* docs(migration): align skill and trajectory assertions * docs: canonicalize trajectory assertion examples
1 parent 9fbdab5 commit 0d9c676

14 files changed

Lines changed: 262 additions & 319 deletions

File tree

apps/web/src/content/docs/docs/next/evaluation/batch-cli.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -111,9 +111,9 @@ JSONL where each line is a JSON object with an `id` matching a test:
111111

112112
The `id` field must match the test `id` for AgentV to route output to the correct grader.
113113

114-
### Output with Tool Trajectory
114+
### Output with Trajectory Assertions
115115

116-
To enable `tool-trajectory` evaluation, include `output` with `tool_calls`:
116+
To enable `trajectory:*` assertions, include `output` with `tool_calls`:
117117

118118
```json
119119
{
@@ -138,7 +138,7 @@ To enable `tool-trajectory` evaluation, include `output` with `tool_calls`:
138138
}
139139
```
140140

141-
AgentV extracts tool calls directly from `output[].tool_calls[]` for `tool-trajectory` graders.
141+
AgentV extracts tool calls directly from `output[].tool_calls[]` for trajectory assertions.
142142

143143
## Grader Implementation
144144

apps/web/src/content/docs/docs/next/evaluation/examples.mdx

Lines changed: 21 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ tests:
124124

125125
See [`examples/features/file-transforms/`](../../../../../examples/features/file-transforms/) for a runnable end-to-end example with a file-producing target and custom grader target.
126126

127-
## Tool Trajectory
127+
## Trajectory Assertions
128128

129129
Validate that an agent uses specific tools during execution:
130130

@@ -139,23 +139,27 @@ tests:
139139
input: Research REST vs GraphQL
140140
assert:
141141
- name: research-check
142-
type: tool-trajectory
143-
mode: any_order
144-
minimums:
145-
knowledgeSearch: 2
146-
documentRetrieve: 1
142+
type: trajectory:tool-used
143+
value:
144+
name: knowledgeSearch
145+
min: 2
146+
- type: trajectory:tool-used
147+
value:
148+
name: documentRetrieve
149+
min: 1
147150
148151
# Validate exact tool sequence
149152
- id: auth-flow
150153
criteria: Agent follows auth sequence
151154
input: Authenticate user
152155
assert:
153156
- name: auth-sequence
154-
type: tool-trajectory
155-
mode: exact
156-
expected:
157-
- tool: checkCredentials
158-
- tool: generateToken
157+
type: trajectory:tool-sequence
158+
value:
159+
mode: exact
160+
steps:
161+
- checkCredentials
162+
- generateToken
159163
```
160164

161165
## Offline Grader Benchmark
@@ -204,11 +208,12 @@ tests:
204208
input: Analyze trace
205209
assert:
206210
- name: trace-check
207-
type: tool-trajectory
208-
mode: in_order
209-
expected:
210-
- tool: webSearch
211-
- tool: readFile
211+
type: trajectory:tool-sequence
212+
value:
213+
mode: in_order
214+
steps:
215+
- webSearch
216+
- readFile
212217
```
213218

214219
## Multi-Turn Conversation

apps/web/src/content/docs/docs/next/evaluation/rubrics.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ assert:
199199
- The agent used Read to inspect the config file before editing
200200
```
201201

202-
This is a lightweight alternative to the `skill-trigger` evaluator when you want to check tool usage with natural-language criteria.
202+
Use this for qualitative tool-use checks. Use `skill-used`, `not-skill-used`, or `trajectory:*` assertions when the requirement can be deterministic.
203203

204204
## Combining with Other Graders
205205

apps/web/src/content/docs/docs/next/graders/tool-trajectory.mdx

Lines changed: 0 additions & 263 deletions
This file was deleted.

0 commit comments

Comments
 (0)