Skip to content

Commit 481fe21

Browse files
haranrkcopybara-github
haranrk
authored and
copybara-github
committed
feat: Add user_simulation ADK eval sample
Add contributing/samples/evaluation/user_simulation/, demonstrating hallucinations_v1 and per_turn_user_simulator_quality_v1 with a dynamically simulated user, run via `adk eval` over the shared home-automation agent. Co-authored-by: Haran Rajkumar <haranrk@google.com> PiperOrigin-RevId: 952613097
1 parent 852a66a commit 481fe21

6 files changed

Lines changed: 192 additions & 0 deletions

File tree

‎contributing/samples/evaluation/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ answer.
4444
| [`custom_metric`](./custom_metric/) | Write your own metric | `temperature_safety_score` (custom) |
4545
| [`llm_judge_match`](./llm_judge_match/) | LLM-judged semantic match | `final_response_match_v2` |
4646
| [`rubric_criteria`](./rubric_criteria/) | LLM-judged quality via rubrics | `rubric_based_final_response_quality_v1`, `rubric_based_tool_use_quality_v1` |
47+
| [`user_simulation`](./user_simulation/) | Dynamically simulated user turns | `hallucinations_v1`, `per_turn_user_simulator_quality_v1` |
4748

4849
## Graph
4950

‎contributing/samples/evaluation/user_simulation/README.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# User simulation
2+
3+
## Overview
4+
5+
Instead of fixed user prompts, an LLM plays the user: it follows a
6+
`conversation_plan` and adopts a `user_persona`, generating each user turn
7+
dynamically in reaction to what the agent says. This tests the agent on
8+
realistic, branching multi-turn dialogue rather than a scripted exchange. For
9+
example, the user starts with a vague goal ("I want my bedroom to be
10+
comfortable.") and only reveals the target temperature once the agent asks for
11+
it, so the agent must actually run the clarification loop.
12+
13+
This sample evaluates the shared home-automation agent with two criteria that
14+
support user simulation:
15+
16+
- `hallucinations_v1`: checks that the agent's responses are grounded in the
17+
tool results and conversation (no invented device states or actions).
18+
- `per_turn_user_simulator_quality_v1`: checks that the *simulated user*
19+
behaved correctly, i.e. it followed the conversation plan and stayed in
20+
persona each turn.
21+
22+
Because both the user simulator and the judges are LLMs, this sample needs a
23+
model credential (a Gemini API key or a Vertex project). Both the simulator and
24+
the judges resolve through the standard model registry, so they run with the
25+
same credentials the agent uses.
26+
27+
## Sample Inputs
28+
29+
`I want my bedroom to be comfortable.`
30+
31+
A NOVICE user who actually wants the bedroom set to 21°C but only reveals the
32+
exact number when the agent asks, and is done once the agent confirms.
33+
34+
`I need to check on my devices.`
35+
36+
An EXPERT user who first asks which devices are on, then asks the agent to turn
37+
off any device that is on in the Living Room, and is done once the agent
38+
confirms.
39+
40+
## How To
41+
42+
Run the sample from the workspace root:
43+
44+
```bash
45+
adk eval contributing/samples/evaluation/home_automation_agent \
46+
contributing/samples/evaluation/user_simulation/home_automation.evalset.json \
47+
--config_file_path contributing/samples/evaluation/user_simulation/eval_config.json \
48+
--print_detailed_results
49+
```
50+
51+
Each eval case in `home_automation.evalset.json` supplies a
52+
`conversation_scenario` instead of a static `conversation` (an `EvalCase` must
53+
have exactly one of the two). Because the user turns are generated at run time,
54+
you'll see a different multi-turn dialogue each run, and the scores will vary
55+
somewhat from run to run; that's expected for LLM-driven simulation and
56+
LLM-as-a-judge scoring.
57+
58+
### `conversation_scenario`
59+
60+
Each scenario describes what the simulated user is trying to do:
61+
62+
- `starting_prompt`: the fixed first user message handed to the agent verbatim.
63+
Every later user turn is generated by the simulator.
64+
- `conversation_plan`: the plan the simulator follows as the conversation plays
65+
out (the goals to accomplish, in order, and any details to reveal only when
66+
asked).
67+
- `user_persona`: the persona the simulator adopts. You can pass one of the
68+
pre-built persona ids and it is resolved from the default persona registry:
69+
- `NOVICE`: relies on the agent for guidance, patient with the agent's
70+
questions, does not correct the agent or troubleshoot its mistakes,
71+
conversational tone.
72+
- `EXPERT`: knows exactly what they want, provides details up front, answers
73+
only relevant questions, corrects the agent's mistakes, professional tone.
74+
- `EVALUATOR`: a third pre-built persona for assessing whether the agent can
75+
accomplish the plan.
76+
77+
### `user_simulator_config`
78+
79+
The eval config's `user_simulator_config` selects and tunes the simulator:
80+
81+
- `type`: the simulator implementation; `llm_backed` uses an LLM to play the
82+
user.
83+
- `model`: the model the simulator uses to generate user turns
84+
(`gemini-2.5-flash` here).
85+
- `max_allowed_invocations`: a safety cap on the number of turns, so a run-off
86+
loop between the agent and the simulated user can't continue forever (the
87+
fixed starting prompt counts as one invocation). Raise it if a scenario needs
88+
more turns to reach its goal; here `8` is plenty.
89+
90+
### Why only certain criteria pair with user simulation
91+
92+
Not every metric works with a dynamically simulated conversation. The two used
93+
here (`hallucinations_v1` and `per_turn_user_simulator_quality_v1`) resolve
94+
their judge model through the standard model registry, so they run with ordinary
95+
Gemini API key or Vertex credentials. `safety_v1` and the `multi_turn_*`
96+
criteria also support user simulation, but they require a Google Cloud / Vertex
97+
project (they call a Vertex-only eval service), so they are omitted here.
98+
99+
### Alternate flow: build an eval set from `conversation_scenarios.json`
100+
101+
Instead of hand-writing the eval set, you can build one from a list of scenarios
102+
plus a shared session input, using the committed `conversation_scenarios.json`
103+
and `session_input.json`:
104+
105+
```bash
106+
adk eval_set create contributing/samples/evaluation/home_automation_agent eval_set_with_scenarios
107+
adk eval_set add_eval_case contributing/samples/evaluation/home_automation_agent eval_set_with_scenarios \
108+
--scenarios_file contributing/samples/evaluation/user_simulation/conversation_scenarios.json \
109+
--session_input_file contributing/samples/evaluation/user_simulation/session_input.json
110+
adk eval contributing/samples/evaluation/home_automation_agent eval_set_with_scenarios \
111+
--config_file_path contributing/samples/evaluation/user_simulation/eval_config.json \
112+
--print_detailed_results
113+
```
114+
115+
Note: `safety_v1` and the `multi_turn_*` criteria also support user simulation
116+
but require a Google Cloud / Vertex project, so they are omitted here.
117+
118+
## Related Guides
119+
120+
- User simulation guide: https://adk.dev/evaluate/user-sim/
121+
- Evaluation overview: https://adk.dev/evaluate/
122+
- Evaluation criteria reference: https://adk.dev/evaluate/criteria/

‎contributing/samples/evaluation/user_simulation/conversation_scenarios.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"scenarios": [
3+
{
4+
"starting_prompt": "I want my bedroom to be comfortable.",
5+
"conversation_plan": "You actually want the bedroom set to 21 degrees Celsius, but only reveal the exact number when the agent asks for it. Once the agent confirms the temperature is set, your goal is complete.",
6+
"user_persona": "NOVICE"
7+
},
8+
{
9+
"starting_prompt": "I need to check on my devices.",
10+
"conversation_plan": "First ask which devices are on. Then ask the agent to turn off any device that is on in the Living Room. Once it confirms, your goal is complete.",
11+
"user_persona": "EXPERT"
12+
}
13+
]
14+
}

‎contributing/samples/evaluation/user_simulation/eval_config.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"criteria": {
3+
"hallucinations_v1": {
4+
"threshold": 0.8,
5+
"judge_model_options": {"judge_model": "gemini-2.5-flash"},
6+
"evaluate_intermediate_nl_responses": true
7+
},
8+
"per_turn_user_simulator_quality_v1": {
9+
"threshold": 0.8,
10+
"judge_model_options": {"judge_model": "gemini-2.5-flash", "num_samples": 5}
11+
}
12+
},
13+
"user_simulator_config": {
14+
"type": "llm_backed",
15+
"model": "gemini-2.5-flash",
16+
"max_allowed_invocations": 8
17+
}
18+
}

‎contributing/samples/evaluation/user_simulation/home_automation.evalset.json‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
{
2+
"eval_set_id": "user_simulation",
3+
"name": "User simulation",
4+
"description": "Dynamically simulated user turns driven by a conversation plan and persona.",
5+
"eval_cases": [
6+
{
7+
"eval_id": "comfortable_bedroom_novice",
8+
"conversation_scenario": {
9+
"starting_prompt": "I want my bedroom to be comfortable.",
10+
"conversation_plan": "You actually want the bedroom set to 21 degrees Celsius, but only reveal the exact number when the agent asks for it. Once the agent confirms the temperature is set, your goal is complete.",
11+
"user_persona": "NOVICE"
12+
},
13+
"session_input": {
14+
"app_name": "home_automation_agent",
15+
"user_id": "user",
16+
"state": {}
17+
}
18+
},
19+
{
20+
"eval_id": "audit_devices_expert",
21+
"conversation_scenario": {
22+
"starting_prompt": "I need to check on my devices.",
23+
"conversation_plan": "First ask which devices are on. Then ask the agent to turn off any device that is on in the Living Room. Once it confirms, your goal is complete.",
24+
"user_persona": "EXPERT"
25+
},
26+
"session_input": {
27+
"app_name": "home_automation_agent",
28+
"user_id": "user",
29+
"state": {}
30+
}
31+
}
32+
]
33+
}

‎contributing/samples/evaluation/user_simulation/session_input.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"app_name": "home_automation_agent",
3+
"user_id": "user"
4+
}

0 commit comments

Comments
 (0)