Skip to content

Commit 57a34ba

Browse files
haranrkcopybara-github
haranrk
authored and
copybara-github
committed
feat: Add test_file_vs_evalset ADK eval sample
Add contributing/samples/evaluation/test_file_vs_evalset/, showing that .test.json and .evalset.json share the same EvalSet schema and both run via `adk eval`: a single-turn .test.json and a multi-session .evalset.json over the shared home-automation agent. Co-authored-by: Haran Rajkumar <haranrk@google.com> PiperOrigin-RevId: 952586726
1 parent fdebd9d commit 57a34ba

5 files changed

Lines changed: 198 additions & 3 deletions

File tree

‎contributing/samples/evaluation/README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,9 +37,10 @@ answer.
3737

3838
## Samples
3939

40-
| Sample | Concept | Criteria |
41-
| ------------------------------------- | -------------------------------------- | --------------------------------------------------- |
42-
| [`basic_criteria`](./basic_criteria/) | Deterministic, reference-based scoring | `tool_trajectory_avg_score`, `response_match_score` |
40+
| Sample | Concept | Criteria |
41+
| ------------------------------------------------- | ------------------------------------------- | --------------------------------------------------- |
42+
| [`basic_criteria`](./basic_criteria/) | Deterministic, reference-based scoring | `tool_trajectory_avg_score`, `response_match_score` |
43+
| [`test_file_vs_evalset`](./test_file_vs_evalset/) | `.test.json` vs `.evalset.json` conventions | `tool_trajectory_avg_score`, `response_match_score` |
4344

4445
## Graph
4546

‎contributing/samples/evaluation/test_file_vs_evalset/README.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Test file vs. eval set
2+
3+
## Overview
4+
5+
A `.test.json` file and a `.evalset.json` file are the same `EvalSet` Pydantic
6+
schema. `adk eval` loads either one with `load_eval_set_from_file`, which
7+
validates by schema, not by file extension, so both run with the exact same
8+
`adk eval` command. The two extensions are only a naming *convention*:
9+
10+
- A `.test.json` is the "unit test" convention: one simple session, kept small and
11+
focused, like a single unit test.
12+
- A `.evalset.json` is the "integration test" convention: multiple, longer,
13+
multi-turn sessions grouped together, like an integration test.
14+
15+
This sample ships one of each against the shared `home_automation_agent`:
16+
17+
- `single_turn.test.json`: a single one-turn session.
18+
- `multi_session.evalset.json`: two sessions, one of which is a two-turn
19+
conversation.
20+
21+
## Sample Inputs
22+
23+
`single_turn.test.json` (one session):
24+
25+
- `What's the temperature in the Kitchen?`
26+
27+
`multi_session.evalset.json` (two sessions):
28+
29+
- `list_then_turn_off` (two turns): `Which devices are on?` then
30+
`Turn that one off.`
31+
- `set_bedroom_temperature` (one turn): `Set the Bedroom to 21 degrees.`
32+
33+
## How To
34+
35+
Both files run with the same `adk eval` command; only the eval-data path changes.
36+
Run from the workspace root.
37+
38+
Run the `.test.json`:
39+
40+
```bash
41+
adk eval contributing/samples/evaluation/home_automation_agent \
42+
contributing/samples/evaluation/test_file_vs_evalset/single_turn.test.json \
43+
--config_file_path contributing/samples/evaluation/test_file_vs_evalset/eval_config.json \
44+
--print_detailed_results
45+
```
46+
47+
Run the `.evalset.json`:
48+
49+
```bash
50+
adk eval contributing/samples/evaluation/home_automation_agent \
51+
contributing/samples/evaluation/test_file_vs_evalset/multi_session.evalset.json \
52+
--config_file_path contributing/samples/evaluation/test_file_vs_evalset/eval_config.json \
53+
--print_detailed_results
54+
```
55+
56+
`--print_detailed_results` prints an Actual-vs-Expected table so you can compare
57+
the agent's real tool calls and responses against the expected values in each
58+
file.
59+
60+
The `.test.json` name is also the format that `pytest` + `AgentEvaluator.evaluate`
61+
auto-discovers, so the same file can be driven from a Python test without change
62+
(not shown here, since this sample uses `adk eval` only).
63+
64+
### `match_type: IN_ORDER`
65+
66+
`eval_config.json` scores the tool trajectory with `match_type: "IN_ORDER"`: the
67+
expected tool calls must appear in the given order, but any extra actual tool
68+
calls in between are tolerated. The `threshold` is `1.0`, so every expected call
69+
(name + args) must still match a real call. `response_match_score` uses a `0.6`
70+
threshold, a ROUGE-1 word-overlap score that tolerates the phrasing variation of
71+
live inference.
72+
73+
## Related Guides
74+
75+
- Evaluation overview: https://adk.dev/evaluate/
76+
- Evaluation criteria reference: https://adk.dev/evaluate/criteria/

‎contributing/samples/evaluation/test_file_vs_evalset/eval_config.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"criteria": {
3+
"tool_trajectory_avg_score": {"threshold": 1.0, "match_type": "IN_ORDER"},
4+
"response_match_score": 0.6
5+
}
6+
}

‎contributing/samples/evaluation/test_file_vs_evalset/multi_session.evalset.json‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
{
2+
"eval_set_id": "multi_session",
3+
"name": "Multi-session eval set",
4+
"description": "Multiple, multi-turn sessions — the '.evalset.json' integration convention.",
5+
"eval_cases": [
6+
{
7+
"eval_id": "list_then_turn_off",
8+
"conversation": [
9+
{
10+
"invocation_id": "ms-1a",
11+
"user_content": {
12+
"parts": [{"text": "Which devices are on?"}],
13+
"role": "user"
14+
},
15+
"final_response": {
16+
"parts": [{"text": "The Living Room device (device_1) is on."}],
17+
"role": "model"
18+
},
19+
"intermediate_data": {
20+
"tool_uses": [{"name": "list_devices", "args": {"status": "ON"}}],
21+
"intermediate_responses": []
22+
}
23+
},
24+
{
25+
"invocation_id": "ms-1b",
26+
"user_content": {
27+
"parts": [{"text": "Turn that one off."}],
28+
"role": "user"
29+
},
30+
"final_response": {
31+
"parts": [{"text": "I have turned off the Living Room device (device_1)."}],
32+
"role": "model"
33+
},
34+
"intermediate_data": {
35+
"tool_uses": [
36+
{"name": "set_device_info",
37+
"args": {"device_id": "device_1", "status": "OFF"}}
38+
],
39+
"intermediate_responses": []
40+
}
41+
}
42+
],
43+
"session_input": {
44+
"app_name": "home_automation_agent",
45+
"user_id": "user",
46+
"state": {}
47+
}
48+
},
49+
{
50+
"eval_id": "set_bedroom_temperature",
51+
"conversation": [
52+
{
53+
"invocation_id": "ms-2a",
54+
"user_content": {
55+
"parts": [{"text": "Set the Bedroom to 21 degrees."}],
56+
"role": "user"
57+
},
58+
"final_response": {
59+
"parts": [{"text": "The Bedroom temperature is now set to 21 degrees Celsius."}],
60+
"role": "model"
61+
},
62+
"intermediate_data": {
63+
"tool_uses": [
64+
{"name": "set_temperature",
65+
"args": {"location": "Bedroom", "temperature": 21}}
66+
],
67+
"intermediate_responses": []
68+
}
69+
}
70+
],
71+
"session_input": {
72+
"app_name": "home_automation_agent",
73+
"user_id": "user",
74+
"state": {}
75+
}
76+
}
77+
]
78+
}

‎contributing/samples/evaluation/test_file_vs_evalset/single_turn.test.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"eval_set_id": "single_turn",
3+
"name": "Single-turn test file",
4+
"description": "One simple session — the '.test.json' unit-test convention.",
5+
"eval_cases": [
6+
{
7+
"eval_id": "kitchen_temperature",
8+
"conversation": [
9+
{
10+
"invocation_id": "tf-1",
11+
"user_content": {
12+
"parts": [{"text": "What's the temperature in the Kitchen?"}],
13+
"role": "user"
14+
},
15+
"final_response": {
16+
"parts": [{"text": "The temperature in the Kitchen is 24 degrees Celsius."}],
17+
"role": "model"
18+
},
19+
"intermediate_data": {
20+
"tool_uses": [
21+
{"name": "get_temperature", "args": {"location": "Kitchen"}}
22+
],
23+
"intermediate_responses": []
24+
}
25+
}
26+
],
27+
"session_input": {
28+
"app_name": "home_automation_agent",
29+
"user_id": "user",
30+
"state": {}
31+
}
32+
}
33+
]
34+
}

0 commit comments

Comments
 (0)