Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions apps/web/src/content/docs/docs/next/evaluation/batch-cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,9 @@ JSONL where each line is a JSON object with an `id` matching a test:

The `id` field must match the test `id` for AgentV to route output to the correct grader.

### Output with Tool Trajectory
### Output with Trajectory Assertions

To enable `tool-trajectory` evaluation, include `output` with `tool_calls`:
To enable `trajectory:*` assertions, include `output` with `tool_calls`:

```json
{
Expand All @@ -138,7 +138,7 @@ To enable `tool-trajectory` evaluation, include `output` with `tool_calls`:
}
```

AgentV extracts tool calls directly from `output[].tool_calls[]` for `tool-trajectory` graders.
AgentV extracts tool calls directly from `output[].tool_calls[]` for trajectory assertions.

## Grader Implementation

Expand Down
37 changes: 21 additions & 16 deletions apps/web/src/content/docs/docs/next/evaluation/examples.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ tests:

See [`examples/features/file-transforms/`](../../../../../examples/features/file-transforms/) for a runnable end-to-end example with a file-producing target and custom grader target.

## Tool Trajectory
## Trajectory Assertions

Validate that an agent uses specific tools during execution:

Expand All @@ -139,23 +139,27 @@ tests:
input: Research REST vs GraphQL
assert:
- name: research-check
type: tool-trajectory
mode: any_order
minimums:
knowledgeSearch: 2
documentRetrieve: 1
type: trajectory:tool-used
value:
name: knowledgeSearch
min: 2
- type: trajectory:tool-used
value:
name: documentRetrieve
min: 1

# Validate exact tool sequence
- id: auth-flow
criteria: Agent follows auth sequence
input: Authenticate user
assert:
- name: auth-sequence
type: tool-trajectory
mode: exact
expected:
- tool: checkCredentials
- tool: generateToken
type: trajectory:tool-sequence
value:
mode: exact
steps:
- checkCredentials
- generateToken
```

## Offline Grader Benchmark
Expand Down Expand Up @@ -204,11 +208,12 @@ tests:
input: Analyze trace
assert:
- name: trace-check
type: tool-trajectory
mode: in_order
expected:
- tool: webSearch
- tool: readFile
type: trajectory:tool-sequence
value:
mode: in_order
steps:
- webSearch
- readFile
```

## Multi-Turn Conversation
Expand Down
2 changes: 1 addition & 1 deletion apps/web/src/content/docs/docs/next/evaluation/rubrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ assert:
- The agent used Read to inspect the config file before editing
```

This is a lightweight alternative to the `skill-trigger` evaluator when you want to check tool usage with natural-language criteria.
Use this for qualitative tool-use checks. Use `skill-used`, `not-skill-used`, or `trajectory:*` assertions when the requirement can be deterministic.

## Combining with Other Graders

Expand Down
263 changes: 0 additions & 263 deletions apps/web/src/content/docs/docs/next/graders/tool-trajectory.mdx

This file was deleted.

Loading
Loading