Merge branch 'google:main' into fix/skill-toolset-instruction-tool-filter

Author: nicobytes

.pre-commit-config.yaml

@@ -72,3 +72,11 @@ repos:
         exclude: (?i)SKILL\.md$
         additional_dependencies:
           - mdformat-gfm
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.4.2
+    hooks:
+      - id: codespell
+        # Configuration (skip globs, ignored words) lives in the
+        # [tool.codespell] table in pyproject.toml.
+        additional_dependencies:
+          - tomli

contributing/samples/evaluation/README.md

@@ -0,0 +1,64 @@
+# ADK evaluation samples
+
+## Overview
+
+A family of single-concept samples that each show one way to evaluate the *same*
+shared home-automation agent with the `adk eval` CLI. Every sample points `adk eval` at one agent and differs only in its eval data and criteria, so you can
+compare evaluation techniques (deterministic reference matching, custom metrics,
+LLM-as-a-judge, rubrics, and user simulation) side by side.
+
+## The shared agent
+
+`home_automation_agent/` is a small agent that controls smart-home devices and
+temperatures. Its five tools (`get_device_info`, `set_device_info`,
+`get_temperature`, `set_temperature`, `list_devices`) are deterministic, backed by
+in-memory state, so eval trajectories are reproducible. The module exposes
+`reset_data()`, which `adk eval` calls to reset that state between eval cases.
+
+Every sample evaluates this same agent. `adk eval` takes the agent path and the
+eval-set path as two separate arguments, so each sub-sample folder holds only
+eval data and its criteria config, never a copy of the agent code.
+
+## How evaluation runs
+
+`adk eval` runs in two phases: (1) live inference, where it actually runs the
+agent against each eval input to produce responses and tool calls, and then (2)
+scoring, where it compares that output against the case's criteria. Because phase 1
+runs the real agent, a model credential is required for every sample, even the
+deterministic ones. Provide a Gemini API key in `home_automation_agent/.env`,
+or configure Vertex.
+Samples that use an LLM judge or a user simulator make additional model calls, but
+they resolve through the same model registry and credentials.
+
+Because live responses vary from run to run, the deterministic, reference-based
+criteria use lenient response thresholds (e.g. `response_match_score` at
+`0.5`) so that harmless phrasing differences don't fail an otherwise-correct
+answer.
+
+## Samples
+
+| Sample                                            | Concept                                     | Criteria                                                                     |
+| ------------------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------- |
+| [`basic_criteria`](./basic_criteria/)             | Deterministic, reference-based scoring      | `tool_trajectory_avg_score`, `response_match_score`                          |
+| [`test_file_vs_evalset`](./test_file_vs_evalset/) | `.test.json` vs `.evalset.json` conventions | `tool_trajectory_avg_score`, `response_match_score`                          |
+| [`custom_metric`](./custom_metric/)               | Write your own metric                       | `temperature_safety_score` (custom)                                          |
+| [`llm_judge_match`](./llm_judge_match/)           | LLM-judged semantic match                   | `final_response_match_v2`                                                    |
+| [`rubric_criteria`](./rubric_criteria/)           | LLM-judged quality via rubrics              | `rubric_based_final_response_quality_v1`, `rubric_based_tool_use_quality_v1` |
+| [`user_simulation`](./user_simulation/)           | Dynamically simulated user turns            | `hallucinations_v1`, `per_turn_user_simulator_quality_v1`                    |
+
+## Graph
+
+```mermaid
+graph TD
+    A[home_automation_agent] --> B(get_device_info)
+    A --> C(set_device_info)
+    A --> D(get_temperature)
+    A --> E(set_temperature)
+    A --> F(list_devices)
+```
+
+## Related Guides
+
+- Evaluation overview: https://adk.dev/evaluate/
+- Evaluation criteria reference: https://adk.dev/evaluate/criteria/
+- User simulation guide: https://adk.dev/evaluate/user-sim/

contributing/samples/evaluation/basic_criteria/README.md

@@ -0,0 +1,78 @@
+# Basic evaluation criteria
+
+## Overview
+
+Evaluates the shared home-automation agent with the two deterministic,
+reference-based criteria:
+
+- `tool_trajectory_avg_score`: does the agent call the right tools with the
+  right args? Each expected tool call (name + args) is compared against what the
+  agent actually did.
+- `response_match_score`: ROUGE-1 word overlap between the agent's final
+  response and a reference answer.
+
+Both criteria are computed locally with no judge model, so this sample needs only
+a model credential for the agent's own inference (a Gemini API key or Vertex).
+
+## Sample Inputs
+
+The eval set (`home_automation.evalset.json`) contains two single-turn cases:
+
+- `Turn off device_2.`
+- `What is the temperature in the Living Room?`
+
+## How To
+
+Run the sample from the workspace root:
+
+```bash
+adk eval contributing/samples/evaluation/home_automation_agent \
+    contributing/samples/evaluation/basic_criteria/home_automation.evalset.json \
+    --config_file_path contributing/samples/evaluation/basic_criteria/eval_config.json \
+    --print_detailed_results
+```
+
+`adk eval` takes the agent folder and the eval-set file as two separate
+arguments, so this folder holds only eval data (`home_automation.evalset.json`),
+the criteria config (`eval_config.json`), and this README, with no agent code.
+
+### `match_type` for tool trajectory
+
+`tool_trajectory_avg_score` has a `match_type` (set to `EXACT` here in
+`eval_config.json`) that controls how the expected and actual tool calls are
+compared:
+
+- `EXACT`: the actual tool calls must match the expected calls one-for-one, in
+  the same order, with identical args. Use this when the trajectory is fully
+  deterministic (as in this sample).
+- `IN_ORDER`: the expected calls must appear in the given order, but extra
+  actual calls in between are tolerated. Useful when the agent may take
+  additional, harmless steps.
+- `ANY_ORDER`: the expected calls must all appear, but order does not matter.
+  Useful when the agent may reorder independent tool calls.
+
+The `threshold` is `1.0`, so every expected call must match for the case to pass.
+
+### Why `response_match_score` uses a `0.6` threshold
+
+`adk eval` runs live inference, so the exact wording of the agent's final
+response varies from run to run (for example, "I have turned off device_2." vs
+"device_2 has been switched off."). `response_match_score` is a ROUGE-1 score,
+which measures word overlap rather than exact-string equality, so it tolerates
+this phrasing variation. The `0.6` threshold requires the response to share most
+of its wording with the reference while still allowing some rewording. Raise it
+toward `1.0` for stricter wording, lower it to tolerate more paraphrasing.
+
+### Expectations captured from a real run
+
+The expected `tool_uses` (tool names and args) in `home_automation.evalset.json`
+were captured from an actual `adk eval` run of the agent: run with
+`--print_detailed_results`, read the printed Actual-vs-Expected, then set the
+expected values to match what the agent really produced. The reference
+`final_response` for each case is an independently written natural answer (not a
+copy of the model output), which is exactly what ROUGE-1 is designed to tolerate.
+
+## Related Guides
+
+- Evaluation overview: https://adk.dev/evaluate/
+- Evaluation criteria reference: https://adk.dev/evaluate/criteria/

contributing/samples/evaluation/basic_criteria/eval_config.json

@@ -0,0 +1,9 @@
+{
+  "criteria": {
+    "tool_trajectory_avg_score": {
+      "threshold": 1.0,
+      "match_type": "EXACT"
+    },
+    "response_match_score": 0.6
+  }
+}

contributing/samples/evaluation/basic_criteria/home_automation.evalset.json

@@ -0,0 +1,62 @@
+{
+  "eval_set_id": "basic_criteria",
+  "name": "Basic deterministic criteria",
+  "description": "Single-turn cases scored by tool trajectory and ROUGE-1.",
+  "eval_cases": [
+    {
+      "eval_id": "turn_off_bedroom_device",
+      "conversation": [
+        {
+          "invocation_id": "basic-1",
+          "user_content": {
+            "parts": [{"text": "Turn off device_2."}],
+            "role": "user"
+          },
+          "final_response": {
+            "parts": [{"text": "device_2 is now turned off."}],
+            "role": "model"
+          },
+          "intermediate_data": {
+            "tool_uses": [
+              {"name": "set_device_info",
+               "args": {"device_id": "device_2", "status": "OFF"}}
+            ],
+            "intermediate_responses": []
+          }
+        }
+      ],
+      "session_input": {
+        "app_name": "home_automation_agent",
+        "user_id": "user",
+        "state": {}
+      }
+    },
+    {
+      "eval_id": "get_living_room_temperature",
+      "conversation": [
+        {
+          "invocation_id": "basic-2",
+          "user_content": {
+            "parts": [{"text": "What is the temperature in the Living Room?"}],
+            "role": "user"
+          },
+          "final_response": {
+            "parts": [{"text": "The Living Room is 22 degrees Celsius."}],
+            "role": "model"
+          },
+          "intermediate_data": {
+            "tool_uses": [
+              {"name": "get_temperature", "args": {"location": "Living Room"}}
+            ],
+            "intermediate_responses": []
+          }
+        }
+      ],
+      "session_input": {
+        "app_name": "home_automation_agent",
+        "user_id": "user",
+        "state": {}
+      }
+    }
+  ]
+}

contributing/samples/evaluation/custom_metric/README.md

@@ -0,0 +1,111 @@
+# Custom evaluation metric
+
+## Overview
+
+This sample shows how to write and register your own evaluation metric when the
+built-in criteria can't express the rule you care about.
+
+`temperature_safety.py` defines `temperature_safety_score`, a metric that
+inspects the agent's *actual* tool calls and fails if any `set_temperature` call
+requests a value outside the safe range of 18-30 Celsius. This is a
+safety/business rule the built-in criteria (`tool_trajectory_avg_score`,
+`response_match_score`, …) can't express, because it checks the *values* passed
+to a specific tool rather than comparing against a reference trajectory.
+
+## Sample Inputs
+
+The eval set (`home_automation.evalset.json`) contains one single-turn case:
+
+- `Set the Bedroom to 21 degrees.`
+
+## How To
+
+### The metric function
+
+A custom metric is any callable with this signature that returns an
+`EvaluationResult`:
+
+```python
+def temperature_safety_score(
+    eval_metric: EvalMetric,
+    actual_invocations: list[Invocation],
+    expected_invocations: Optional[list[Invocation]],
+    conversation_scenario: Optional[ConversationScenario],
+) -> EvaluationResult:
+```
+
+The function may be sync or async. Inside it:
+
+- Read the agent's actual tool calls for each invocation with
+  `get_all_tool_calls(invocation.intermediate_data)`. This returns
+  `google.genai.types.FunctionCall` objects, each with a `.name` and `.args`, so
+  you can inspect exactly what the agent called and with which arguments.
+- Return an `EvaluationResult`. Set `overall_eval_status` (PASSED/FAILED) and a
+  matching `per_invocation_results` entry for every invocation. `adk eval`
+  derives pass/fail from the status, not from `overall_score` alone. A metric
+  that sets only a score leaves the status at `NOT_EVALUATED`, and the case is
+  reported as not passed even with a perfect score. (When the status is not
+  `NOT_EVALUATED`, `adk eval` also requires one `per_invocation_results` entry
+  per invocation.)
+
+### Registering the metric
+
+The metric is wired in via `custom_metrics` in `eval_config.json`:
+
+```json
+{
+  "criteria": {
+    "temperature_safety_score": 1.0
+  },
+  "custom_metrics": {
+    "temperature_safety_score": {
+      "code_config": {"name": "temperature_safety.temperature_safety_score"},
+      "description": "Fails if any set_temperature call is outside 18-30 Celsius."
+    }
+  }
+}
+```
+
+The metric name appears in both `criteria` (with its threshold) and
+`custom_metrics`. The `code_config.name` is a dotted path: everything before the
+last dot is the *module* (`temperature_safety`) and the last segment is the
+*function* (`temperature_safety_score`).
+
+### Running the sample
+
+`adk eval` resolves `code_config.name` by calling
+`importlib.import_module("temperature_safety")`, which searches `sys.path`. The
+metric module lives in this sample folder, which is not on `sys.path` by default,
+so put the folder on `PYTHONPATH` when you run the eval:
+
+```bash
+PYTHONPATH=contributing/samples/evaluation/custom_metric \
+adk eval contributing/samples/evaluation/home_automation_agent \
+    contributing/samples/evaluation/custom_metric/home_automation.evalset.json \
+    --config_file_path contributing/samples/evaluation/custom_metric/eval_config.json \
+    --print_detailed_results
+```
+
+Run it from the workspace root. Without the `PYTHONPATH` prefix you'll get
+`ImportError: Could not import custom metric function ...`.
+
+The shipped case sets the Bedroom to a valid 21 Celsius, so the metric scores
+1.0 (PASSED):
+
+```
+custom_metric:
+  Tests passed: 1
+  Tests failed: 0
+...
+Metric: temperature_safety_score, Status: PASSED, Score: 1.0, Threshold: 1.0
+```
+
+An unsafe value (for example, `set_temperature(location="Bedroom", temperature=45)`) would score 0.0 (FAILED). We keep the agent well-behaved
+and demonstrate the passing path rather than forcing an unsafe call from live
+inference; the FAIL branch is exactly the `18 <= temperature <= 30` check in
+`temperature_safety.py`.
+
+## Related Guides
+
+- Evaluation overview: https://adk.dev/evaluate/
+- Evaluation criteria reference: https://adk.dev/evaluate/criteria/

contributing/samples/evaluation/custom_metric/eval_config.json

@@ -0,0 +1,11 @@
+{
+  "criteria": {
+    "temperature_safety_score": 1.0
+  },
+  "custom_metrics": {
+    "temperature_safety_score": {
+      "code_config": {"name": "temperature_safety.temperature_safety_score"},
+      "description": "Fails if any set_temperature call is outside 18-30 Celsius."
+    }
+  }
+}

contributing/samples/evaluation/custom_metric/home_automation.evalset.json

@@ -0,0 +1,35 @@
+{
+  "eval_set_id": "custom_metric",
+  "name": "Custom metric: temperature safety",
+  "description": "A valid set_temperature call; the custom metric scores 1.0.",
+  "eval_cases": [
+    {
+      "eval_id": "set_bedroom_safe_temp",
+      "conversation": [
+        {
+          "invocation_id": "cm-1",
+          "user_content": {
+            "parts": [{"text": "Set the Bedroom to 21 degrees."}],
+            "role": "user"
+          },
+          "final_response": {
+            "parts": [{"text": "Temperature in Bedroom set to 21C."}],
+            "role": "model"
+          },
+          "intermediate_data": {
+            "tool_uses": [
+              {"name": "set_temperature",
+               "args": {"location": "Bedroom", "temperature": 21}}
+            ],
+            "intermediate_responses": []
+          }
+        }
+      ],
+      "session_input": {
+        "app_name": "home_automation_agent",
+        "user_id": "user",
+        "state": {}
+      }
+    }
+  ]
+}