You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: RELEASE_NOTES.md
+17Lines changed: 17 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,6 +2,23 @@
2
2
3
3
All notable changes to this project will be documented in this file.
4
4
5
+
## [0.3.0] - 2026-05-29
6
+
### Breaking
7
+
- The eval data format changed from ADK `EvalSet` to Vertex AI `EvaluationDataset`. Existing `tests/eval/evalsets/*.evalset.json` files are no longer read by `agents-cli eval generate` and friends. See [Migrating Eval Datasets](docs/src/reference/eval-dataset-migration.md) for the conversion. `scaffold upgrade` now prints a notice when legacy files are detected.
8
+
9
+
### Eval - Quality Flywheel (preview)
10
+
- Added `eval dataset synthesize` for LLM-driven user-simulation dataset generation.
11
+
- Added `eval generate` to run agent inference over an `EvaluationDataset` and emit traces.
12
+
- Added `eval grade` to score agent traces against built-in or custom metrics.
13
+
- Added `eval submit` to submit an end-to-end cloud-side evaluation run on Vertex AI Eval Service.
14
+
- Added `eval results` to fetch results from a completed cloud evaluation run.
15
+
- Added `eval analyze` for failure-mode analysis over graded results.
16
+
- Added `eval metric list` to discover built-in evaluation metrics.
17
+
- Rewrote the `eval` skill end-to-end to cover the Quality Flywheel workflow (dataset, generate, grade, analyze, optimize).
Copy file name to clipboardExpand all lines: docs/src/guide/development.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -76,15 +76,16 @@ Add and remove dependencies with [uv](https://docs.astral.sh/uv/):
76
76
77
77
## Phase 3: Evaluate
78
78
79
-
Run structured evaluations to validate agent behavior. This uses the [ADK eval CLI](https://google.github.io/adk-docs/evaluate/) under the hood.
79
+
Run structured evaluations to validate agent behavior. This uses the [GenAI Eval SDK](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/agent-evaluation) under the hood.
80
80
81
81
```bash
82
-
agents-cli eval run
82
+
agents-cli eval generate
83
+
agents-cli eval grade
83
84
```
84
85
85
86
Expect **5-10+ iterations** of the eval-fix loop before your agent consistently passes. Start with 1-2 core eval cases, fix failures, then expand coverage.
86
87
87
-
See the [Evaluation Guide](evaluation.md) for metrics, evalset schemas, and the full methodology.
88
+
See the [Evaluation Guide](evaluation.md) for metrics, dataset schemas, and the full methodology.
Run structured evaluations to confirm your agent calls the right tools, produces quality responses, and handles edge cases. Under the hood, `agents-cli eval run` uses the [ADK eval CLI](https://google.github.io/adk-docs/evaluate/) to run evaluations with LLM-as-judge scoring.
3
+
Run structured evaluations to confirm your agent calls the right tools, produces quality responses, and handles edge cases. Under the hood, evaluation uses the [Gemini Enterprise Agent Platform GenAI Eval SDK](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/agent-evaluation) to grade evaluations.
4
+
5
+
!!! note "Upgrading from an older agents-cli?"
6
+
If your project still has `tests/eval/evalsets/*.evalset.json` files from a previous version, see [Migrating Eval Datasets](../reference/eval-dataset-migration.md) for the new format.
4
7
5
8
---
6
9
7
10
## Run Your First Evaluation
8
11
9
-
Your project includes a default eval set at `tests/eval/evalsets/basic.evalset.json` and scoring criteria at `tests/eval/eval_config.json`. Run it:
12
+
Your project includes a default dataset at `tests/eval/datasets/basic-dataset.json` and metrics configuration `tests/eval/eval_config.yaml`. Run it:
10
13
11
14
```bash
12
-
agents-cli eval run
15
+
agents-cli eval generate
16
+
agents-cli eval grade
13
17
```
14
18
15
-
The output shows scores for each eval case against the configured rubrics. A score above the threshold (default: 0.8) passes.
19
+
The output shows scores for each eval case against the configured metrics.
16
20
17
21
```bash
18
-
# Run a specific eval set
19
-
agents-cli eval run --evalset tests/eval/evalsets/custom.evalset.json
For full documentation on eval case schemas, available metrics, rubric writing, and multi-turn testing, see the [ADK Evaluation Guide](https://google.github.io/adk-docs/evaluate/).
31
+
For full documentation on eval case schemas and available metrics, see the [Gemini Enterprise Agent Platform Evaluation documentation](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/agent-evaluation).
32
+
33
+
### Available Metrics Reference
30
34
31
-
Quick reference for choosing metrics:
35
+
You can choose from a wide range of built-in metrics depending on your agent's capabilities and the task at hand. To see the full list of available metrics, run:
32
36
33
-
-**Agents with custom function tools** — use `tool_trajectory_avg_score` + `rubric_based_final_response_quality_v1`.
34
-
-**Agents with `google_search` or model-internal tools** — use only `rubric_based_final_response_quality_v1` (model-internal tools don't appear in trajectory).
35
-
-**RAG agents** — use `rubric_based_final_response_quality_v1` + `hallucinations_v1`.
37
+
```bash
38
+
agents-cli eval metric list
39
+
```
40
+
41
+
#### Common metrics at a glance
42
+
43
+
A short reference for the most-used built-in metric IDs. Use `agents-cli eval metric list` for the full set with descriptions.
44
+
45
+
| Metric ID | What it grades |
46
+
|---|---|
47
+
|`general_quality`| Overall response quality with auto-generated content-based criteria. Recommended starting point for non-agent eval. |
48
+
|`text_quality`| Linguistic aspects: fluency, coherence, grammar. |
49
+
|`instruction_following`| How well the response adheres to specific constraints and instructions. |
Each custom metric must conform to either the **Code Execution Metric** or **LLM-as-a-Judge Metric** (`LLMMetric`) schema:
94
+
- **Code Execution Metric**: Used to run custom Python code for evaluation. Must have a `name` and a `custom_function` (containing a `def evaluate(instance):` signature). By default, the function executes **locally in the CLI process** — no GCP project or region is required, but the user-supplied code runs with the CLI's privileges. Add `"execution": "remote"` to opt into Vertex AI's sandboxed `CodeExecutionMetric` (server-side), which requires a configured GCP project + region.
95
+
- **LLM-as-a-Judge Metric**: Used to evaluate responses using an LLM judge. Must have a `name` and a `prompt_template`. Optional fields include `rubric_group_name`, `judge_model` (e.g., `gemini-flash-latest`), and `judge_model_sampling_count` (between `1` and `32`).
96
+
97
+
### Quick Reference for Common Scenarios
98
+
99
+
- **Agents with custom function tools** — Use `tool_use_quality` (for single-turn) or `multi_turn_tool_use_quality` + `multi_turn_trajectory_quality` (for multi-turn).
100
+
- **RAG agents** — Use `grounding` + `hallucination` + `safety`.
101
+
- **Conversational assistants** — Use `general_quality` or `multi_turn_general_quality`.
102
+
- **Goal-oriented agents** — Use `multi_turn_task_success`.
36
103
37
104
---
38
105
@@ -41,10 +108,60 @@ Quick reference for choosing metrics:
41
108
Evaluation is iterative. Expect 5-10+ cycles before your agent consistently passes.
42
109
43
110
1. **Write 1-2 core eval cases** covering the most important behavior.
44
-
2.**Run**: `agents-cli eval run`
111
+
2. **Run**: `agents-cli eval generate`followed by `agents-cli eval grade`
45
112
3. **Read the results** — which cases failed and why.
46
113
4. **Fix** — adjust the agent's instruction, tools, or logic.
6. **Expand** — once core cases pass, add edge cases and new scenarios.
49
116
50
-
For full documentation on eval case schemas, metrics, rubrics, and user simulation, see the [ADK Evaluation Guide](https://google.github.io/adk-docs/evaluate/).
117
+
---
118
+
119
+
## Beyond `generate` and `grade`
120
+
121
+
`generate`and `grade` form the inner loop, but the eval surface has a few more commands worth knowing about. Each is a separate step you reach for as your eval setup matures.
122
+
123
+
### `agents-cli eval dataset synthesize`
124
+
125
+
Bootstraps a dataset by inspecting your local ADK agent and generating multi-turn conversation scenarios for it — no input file required. Useful for cold-starting evaluation on a new agent or expanding coverage without writing every case by hand. Each generated case includes a starting user message, a conversation plan, and the full agent trace produced by playing the scenario out against an LLM-backed user simulator.
126
+
127
+
```bash
128
+
agents-cli eval dataset synthesize --count 10
129
+
```
130
+
131
+
Steer what gets generated with `--instruction` (e.g. `"Scenarios where the user changes their mind"`) and `--environment-context` (e.g. `"Today is Monday. Flights to Paris are available."`). The output is a regular `*-dataset.json` file you can edit, commit, and feed back into `eval grade` directly (the trace is already populated, so you can skip `eval generate`).
132
+
133
+
### `agents-cli eval compare`
134
+
135
+
Compares two grade results side by side so you can see whether a change actually improved things.
A typical use is comparing a "before fix" run against an "after fix" run during the eval-fix loop.
142
+
143
+
### `agents-cli eval analyze`
144
+
145
+
Clusters failure modes from a grade-results file into themes, so you can see *what kinds of things* are going wrong instead of skimming individual cases.
Prints every built-in metric the SDK supports, with a short description for each. The starting point when you want to know what's available beyond the common metrics table above.
154
+
155
+
### `agents-cli eval optimize`
156
+
157
+
Once your evals are in place, `eval optimize` uses them to automatically tune your agent's prompts.
158
+
159
+
```bash
160
+
agents-cli eval optimize
161
+
```
162
+
163
+
A run takes anywhere from a few minutes to several hours depending on dataset size and metric complexity, so it's not something to run over and over. Reach for it after simpler approaches (rewriting the prompt yourself, adjusting metrics, fixing failing cases by hand) have run their course.
164
+
165
+
---
166
+
167
+
For full documentation on eval case schemas, metrics, and user simulation, see the [Gemini Enterprise Agent Platform Evaluation documentation](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/agent-evaluation).
"parts": [{"text": "Hello, what can you help me with?"}]
182
-
}
183
-
}
184
-
],
185
-
"session_input": {
186
-
"app_name": "app",
187
-
"user_id": "eval_user",
188
-
"state": {}
175
+
"eval_case_id": "greeting",
176
+
"prompt": {
177
+
"role": "user",
178
+
"parts": [{"text": "Hello, what can you help me with?"}]
189
179
}
190
180
}
191
181
]
192
182
}
193
183
```
194
184
195
-
Each eval case defines a user message and session context. The eval system sends the message to your agent and scores the response using LLM-as-judge criteria defined in `eval_config.json`.
185
+
Each eval case defines a user message. The evaluation system sends the message to your agent and grades the response using metrics specified in `eval_config.yaml`.
196
186
197
187
Run it:
198
188
199
189
```bash
200
-
agents-cli eval run
190
+
agents-cli eval generate
191
+
agents-cli eval grade
201
192
```
202
193
203
-
The output shows scores for each eval case against the configured rubrics (relevance, helpfulness). A score above the threshold (default: 0.8) passes.
194
+
The output shows scores for each eval case against the configured metrics.
204
195
205
-
For the full evaluation workflow — writing test cases, adding metrics, the eval-fix loop — see the [Evaluation Guide](evaluation.md).
196
+
For the full evaluation workflow — writing test cases, adding metrics, the eval-fix loop, and the rest of the eval surface (`eval dataset synthesize`, `eval compare`, `eval analyze`, `eval metric list`, and `eval optimize`) — see the [Evaluation Guide](evaluation.md).
206
197
207
198
---
208
199
@@ -258,7 +249,7 @@ See the [Observability Guide](observability/index.md) for verification steps, fu
258
249
|`agents-cli playground`| Started the ADK playground for interactive testing |
259
250
|`agents-cli run "..."`| Tested the agent from the terminal |
260
251
| Edited `agent.py`| Customized the persona and added a tool |
261
-
|`agents-cli eval run`| Validated agent behavior with structured evaluations |
252
+
|`agents-cli eval generate` followed by `agents-cli eval grade`| Validated agent behavior with structured evaluations |
262
253
|`agents-cli deploy`| Deployed the agent to Google Cloud |
263
254
| Trace explorer + content logs | Verified tracing and set up prompt-response logging |
0 commit comments