Skip to content

Commit 852a66a

Browse files
haranrkcopybara-github
haranrk
authored and
copybara-github
committed
feat: Add rubric_criteria ADK eval sample
Add contributing/samples/evaluation/rubric_criteria/, demonstrating rubric_based_final_response_quality_v1 and rubric_based_tool_use_quality_v1 run via `adk eval` over the shared home-automation agent. Co-authored-by: Haran Rajkumar <haranrk@google.com> PiperOrigin-RevId: 952612418
1 parent 0f0fa6b commit 852a66a

4 files changed

Lines changed: 187 additions & 6 deletions

File tree

‎contributing/samples/evaluation/README.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -37,12 +37,13 @@ answer.
3737

3838
## Samples
3939

40-
| Sample | Concept | Criteria |
41-
| ------------------------------------------------- | ------------------------------------------- | --------------------------------------------------- |
42-
| [`basic_criteria`](./basic_criteria/) | Deterministic, reference-based scoring | `tool_trajectory_avg_score`, `response_match_score` |
43-
| [`test_file_vs_evalset`](./test_file_vs_evalset/) | `.test.json` vs `.evalset.json` conventions | `tool_trajectory_avg_score`, `response_match_score` |
44-
| [`custom_metric`](./custom_metric/) | Write your own metric | `temperature_safety_score` (custom) |
45-
| [`llm_judge_match`](./llm_judge_match/) | LLM-judged semantic match | `final_response_match_v2` |
40+
| Sample | Concept | Criteria |
41+
| ------------------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------- |
42+
| [`basic_criteria`](./basic_criteria/) | Deterministic, reference-based scoring | `tool_trajectory_avg_score`, `response_match_score` |
43+
| [`test_file_vs_evalset`](./test_file_vs_evalset/) | `.test.json` vs `.evalset.json` conventions | `tool_trajectory_avg_score`, `response_match_score` |
44+
| [`custom_metric`](./custom_metric/) | Write your own metric | `temperature_safety_score` (custom) |
45+
| [`llm_judge_match`](./llm_judge_match/) | LLM-judged semantic match | `final_response_match_v2` |
46+
| [`rubric_criteria`](./rubric_criteria/) | LLM-judged quality via rubrics | `rubric_based_final_response_quality_v1`, `rubric_based_tool_use_quality_v1` |
4647

4748
## Graph
4849

‎contributing/samples/evaluation/rubric_criteria/README.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Rubric-based evaluation
2+
3+
## Overview
4+
5+
Score response quality and tool-use quality against custom yes/no rubrics judged
6+
by an LLM, with no reference answer required. This sample evaluates the shared
7+
home-automation agent with two reference-free, LLM-judged criteria:
8+
9+
- `rubric_based_final_response_quality_v1`: judges the agent's final response
10+
against rubrics about *how good the answer is* (does it name the device(s) in
11+
the requested location and report each one's on/off status? is it concise?).
12+
- `rubric_based_tool_use_quality_v1`: judges the agent's tool calls against
13+
rubrics about *how it used its tools* (does it filter `list_devices` by the
14+
room the user named? does it avoid changing the temperature when only asked to
15+
inspect devices?).
16+
17+
Each criterion scores the agent against the custom yes/no rubrics you provide, so
18+
you evaluate quality directly instead of matching a golden answer. Because the
19+
judge is an LLM, this sample needs a model credential for the judge (a Gemini API
20+
key or Vertex), in addition to the credential used for the agent's own inference.
21+
22+
## Sample Inputs
23+
24+
The eval set (`home_automation.evalset.json`) contains one single-turn case:
25+
26+
- `What devices are in the Bedroom?`
27+
28+
## How To
29+
30+
Run the sample from the workspace root:
31+
32+
```bash
33+
adk eval contributing/samples/evaluation/home_automation_agent \
34+
contributing/samples/evaluation/rubric_criteria/home_automation.evalset.json \
35+
--config_file_path contributing/samples/evaluation/rubric_criteria/eval_config.json \
36+
--print_detailed_results
37+
```
38+
39+
`adk eval` takes the agent folder and the eval-set file as two separate
40+
arguments, so this folder holds only eval data (`home_automation.evalset.json`),
41+
the criteria config (`eval_config.json`), and this README, with no agent code.
42+
43+
### The `rubrics` list
44+
45+
Both criteria are configured in `eval_config.json` through a `rubrics` list. Each
46+
rubric is a single yes/no property the judge decides against the agent's
47+
behavior:
48+
49+
- `rubric_id`: a stable, unique identifier for the rubric (e.g.
50+
`reports_device_state`). It labels the rubric in the scored output and must be
51+
unique within the criterion.
52+
- `rubric_content.text_property`: the natural-language property being judged,
53+
phrased so the answer is a clean "yes" or "no" (e.g. "The response is concise
54+
and free of filler."). Write each property as one fair, achievable behavior;
55+
avoid bundling several requirements into one rubric.
56+
57+
The `rubrics` list must be non-empty: `RubricBasedEvaluator` asserts this at
58+
init, so a rubric-based criterion with no rubrics fails immediately.
59+
60+
For each invocation the judge is sampled `num_samples` times (here `5`); the
61+
per-rubric verdicts are combined by majority vote, and the criterion score is the
62+
fraction of rubrics that pass. The `threshold` then decides the case:
63+
`rubric_based_final_response_quality_v1` uses `0.8` (a strong majority of its
64+
rubrics must hold), and `rubric_based_tool_use_quality_v1` uses `1.0` (every
65+
tool-use rubric must hold).
66+
67+
### Criterion-level vs. per-case rubrics
68+
69+
The rubrics in `eval_config.json` are criterion-level: they apply to every
70+
eval case scored by that criterion. You can also attach rubrics to a single case
71+
via `EvalCase.rubrics` in the eval set. Per-case rubrics are filtered by their
72+
`type` field before they are handed to a criterion:
73+
74+
- `rubric_based_final_response_quality_v1` only consumes rubrics of type
75+
`FINAL_RESPONSE_QUALITY`.
76+
- `rubric_based_tool_use_quality_v1` only consumes rubrics of type
77+
`TOOL_USE_QUALITY`.
78+
79+
The filtered per-case rubrics are then added to the criterion-level list to
80+
form the effective rubric list for the case. Rubric IDs must be unique across the
81+
two scopes: a `rubric_id` that appears in both the criterion-level list and a
82+
case's `EvalCase.rubrics` raises an error: duplicates are not silently
83+
deduplicated or overridden. Use criterion-level rubrics for expectations shared
84+
across the whole eval set and per-case rubrics for expectations unique to one
85+
scenario.
86+
87+
### When quality rubrics beat reference matching
88+
89+
Reach for rubric-based criteria when "correct" isn't a single golden answer or
90+
trajectory. Reference-based criteria like `response_match_score` (ROUGE-1) or
91+
`tool_trajectory_avg_score` require you to write the expected answer or the exact
92+
sequence of tool calls, and they penalize any legitimate variation: a
93+
reworded-but-correct answer, or a harmless extra tool call. Rubrics instead let
94+
you state the *qualities* that matter ("confirms the device and its state", "uses
95+
a location filter") and let the judge decide whether the agent exhibited them,
96+
regardless of exact wording or an extra step. That makes them a good fit for
97+
open-ended responses and flexible trajectories where you care about quality, not
98+
byte-for-byte equality. The trade-off is the usual LLM-judge cost: a model call
99+
per sample, plus some run-to-run variability that `num_samples` and majority vote
100+
are there to smooth out. Keep the deterministic reference-based criteria when the
101+
answer or trajectory really is fixed.
102+
103+
## Related Guides
104+
105+
- Evaluation overview: https://adk.dev/evaluate/
106+
- Evaluation criteria reference: https://adk.dev/evaluate/criteria/

‎contributing/samples/evaluation/rubric_criteria/eval_config.json‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{
2+
"criteria": {
3+
"rubric_based_final_response_quality_v1": {
4+
"threshold": 0.8,
5+
"judge_model_options": {"judge_model": "gemini-2.5-flash", "num_samples": 5},
6+
"rubrics": [
7+
{
8+
"rubric_id": "reports_device_state",
9+
"rubric_content": {
10+
"text_property": "The response names the specific device(s) in the requested location and states each one's current on/off status."
11+
}
12+
},
13+
{
14+
"rubric_id": "concise",
15+
"rubric_content": {
16+
"text_property": "The response is concise and free of filler."
17+
}
18+
}
19+
]
20+
},
21+
"rubric_based_tool_use_quality_v1": {
22+
"threshold": 1.0,
23+
"judge_model_options": {"judge_model": "gemini-2.5-flash", "num_samples": 5},
24+
"rubrics": [
25+
{
26+
"rubric_id": "uses_location_filter",
27+
"rubric_content": {
28+
"text_property": "When the user names a room, the agent calls list_devices with a matching location filter."
29+
}
30+
},
31+
{
32+
"rubric_id": "no_unrequested_temperature_change",
33+
"rubric_content": {
34+
"text_property": "The agent does not call set_temperature when the user only asked to list or inspect devices."
35+
}
36+
}
37+
]
38+
}
39+
}
40+
}

‎contributing/samples/evaluation/rubric_criteria/home_automation.evalset.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"eval_set_id": "rubric_criteria",
3+
"name": "Rubric-based quality",
4+
"description": "LLM-judged final-response and tool-use quality against custom rubrics.",
5+
"eval_cases": [
6+
{
7+
"eval_id": "list_bedroom_devices",
8+
"conversation": [
9+
{
10+
"invocation_id": "rc-1",
11+
"user_content": {
12+
"parts": [{"text": "What devices are in the Bedroom?"}],
13+
"role": "user"
14+
},
15+
"final_response": {
16+
"parts": [{"text": "The Bedroom has device_2, which is off."}],
17+
"role": "model"
18+
},
19+
"intermediate_data": {
20+
"tool_uses": [
21+
{"name": "list_devices", "args": {"location": "Bedroom"}}
22+
],
23+
"intermediate_responses": []
24+
}
25+
}
26+
],
27+
"session_input": {
28+
"app_name": "home_automation_agent",
29+
"user_id": "user",
30+
"state": {}
31+
}
32+
}
33+
]
34+
}

0 commit comments

Comments
 (0)