feat: Add custom_metric ADK eval sample

Add contributing/samples/evaluation/custom_metric/, demonstrating a
user-defined temperature_safety_score metric plugged into `adk eval` over the
shared home-automation agent.

Co-authored-by: Haran Rajkumar <haranrk@google.com>
PiperOrigin-RevId: 952602145

Author: haranrk

contributing/samples/evaluation/README.md

@@ -41,6 +41,7 @@ answer.
 | ------------------------------------------------- | ------------------------------------------- | --------------------------------------------------- |
 | [`basic_criteria`](./basic_criteria/)             | Deterministic, reference-based scoring      | `tool_trajectory_avg_score`, `response_match_score` |
 | [`test_file_vs_evalset`](./test_file_vs_evalset/) | `.test.json` vs `.evalset.json` conventions | `tool_trajectory_avg_score`, `response_match_score` |
+| [`custom_metric`](./custom_metric/)               | Write your own metric                       | `temperature_safety_score` (custom)                 |
 
 ## Graph
 

contributing/samples/evaluation/custom_metric/README.md

@@ -0,0 +1,111 @@
+# Custom evaluation metric
+
+## Overview
+
+This sample shows how to write and register your own evaluation metric when the
+built-in criteria can't express the rule you care about.
+
+`temperature_safety.py` defines `temperature_safety_score`, a metric that
+inspects the agent's *actual* tool calls and fails if any `set_temperature` call
+requests a value outside the safe range of 18-30 Celsius. This is a
+safety/business rule the built-in criteria (`tool_trajectory_avg_score`,
+`response_match_score`, …) can't express, because it checks the *values* passed
+to a specific tool rather than comparing against a reference trajectory.
+
+## Sample Inputs
+
+The eval set (`home_automation.evalset.json`) contains one single-turn case:
+
+- `Set the Bedroom to 21 degrees.`
+
+## How To
+
+### The metric function
+
+A custom metric is any callable with this signature that returns an
+`EvaluationResult`:
+
+```python
+def temperature_safety_score(
+    eval_metric: EvalMetric,
+    actual_invocations: list[Invocation],
+    expected_invocations: Optional[list[Invocation]],
+    conversation_scenario: Optional[ConversationScenario],
+) -> EvaluationResult:
+```
+
+The function may be sync or async. Inside it:
+
+- Read the agent's actual tool calls for each invocation with
+  `get_all_tool_calls(invocation.intermediate_data)`. This returns
+  `google.genai.types.FunctionCall` objects, each with a `.name` and `.args`, so
+  you can inspect exactly what the agent called and with which arguments.
+- Return an `EvaluationResult`. Set `overall_eval_status` (PASSED/FAILED) and a
+  matching `per_invocation_results` entry for every invocation. `adk eval`
+  derives pass/fail from the status, not from `overall_score` alone. A metric
+  that sets only a score leaves the status at `NOT_EVALUATED`, and the case is
+  reported as not passed even with a perfect score. (When the status is not
+  `NOT_EVALUATED`, `adk eval` also requires one `per_invocation_results` entry
+  per invocation.)
+
+### Registering the metric
+
+The metric is wired in via `custom_metrics` in `eval_config.json`:
+
+```json
+{
+  "criteria": {
+    "temperature_safety_score": 1.0
+  },
+  "custom_metrics": {
+    "temperature_safety_score": {
+      "code_config": {"name": "temperature_safety.temperature_safety_score"},
+      "description": "Fails if any set_temperature call is outside 18-30 Celsius."
+    }
+  }
+}
+```
+
+The metric name appears in both `criteria` (with its threshold) and
+`custom_metrics`. The `code_config.name` is a dotted path: everything before the
+last dot is the *module* (`temperature_safety`) and the last segment is the
+*function* (`temperature_safety_score`).
+
+### Running the sample
+
+`adk eval` resolves `code_config.name` by calling
+`importlib.import_module("temperature_safety")`, which searches `sys.path`. The
+metric module lives in this sample folder, which is not on `sys.path` by default,
+so put the folder on `PYTHONPATH` when you run the eval:
+
+```bash
+PYTHONPATH=contributing/samples/evaluation/custom_metric \
+adk eval contributing/samples/evaluation/home_automation_agent \
+    contributing/samples/evaluation/custom_metric/home_automation.evalset.json \
+    --config_file_path contributing/samples/evaluation/custom_metric/eval_config.json \
+    --print_detailed_results
+```
+
+Run it from the workspace root. Without the `PYTHONPATH` prefix you'll get
+`ImportError: Could not import custom metric function ...`.
+
+The shipped case sets the Bedroom to a valid 21 Celsius, so the metric scores
+1.0 (PASSED):
+
+```
+custom_metric:
+  Tests passed: 1
+  Tests failed: 0
+...
+Metric: temperature_safety_score, Status: PASSED, Score: 1.0, Threshold: 1.0
+```
+
+An unsafe value (for example, `set_temperature(location="Bedroom", temperature=45)`) would score 0.0 (FAILED). We keep the agent well-behaved
+and demonstrate the passing path rather than forcing an unsafe call from live
+inference; the FAIL branch is exactly the `18 <= temperature <= 30` check in
+`temperature_safety.py`.
+
+## Related Guides
+
+- Evaluation overview: https://adk.dev/evaluate/
+- Evaluation criteria reference: https://adk.dev/evaluate/criteria/

contributing/samples/evaluation/custom_metric/eval_config.json

@@ -0,0 +1,11 @@
+{
+  "criteria": {
+    "temperature_safety_score": 1.0
+  },
+  "custom_metrics": {
+    "temperature_safety_score": {
+      "code_config": {"name": "temperature_safety.temperature_safety_score"},
+      "description": "Fails if any set_temperature call is outside 18-30 Celsius."
+    }
+  }
+}

contributing/samples/evaluation/custom_metric/home_automation.evalset.json

@@ -0,0 +1,35 @@
+{
+  "eval_set_id": "custom_metric",
+  "name": "Custom metric: temperature safety",
+  "description": "A valid set_temperature call; the custom metric scores 1.0.",
+  "eval_cases": [
+    {
+      "eval_id": "set_bedroom_safe_temp",
+      "conversation": [
+        {
+          "invocation_id": "cm-1",
+          "user_content": {
+            "parts": [{"text": "Set the Bedroom to 21 degrees."}],
+            "role": "user"
+          },
+          "final_response": {
+            "parts": [{"text": "Temperature in Bedroom set to 21C."}],
+            "role": "model"
+          },
+          "intermediate_data": {
+            "tool_uses": [
+              {"name": "set_temperature",
+               "args": {"location": "Bedroom", "temperature": 21}}
+            ],
+            "intermediate_responses": []
+          }
+        }
+      ],
+      "session_input": {
+        "app_name": "home_automation_agent",
+        "user_id": "user",
+        "state": {}
+      }
+    }
+  ]
+}

contributing/samples/evaluation/custom_metric/temperature_safety.py

@@ -0,0 +1,87 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""A custom eval metric: fail if any set_temperature call is unsafe.
+
+A custom metric is any callable with this signature that returns an
+EvaluationResult. It is wired into an eval via `custom_metrics` in the
+EvalConfig (see eval_config.json).
+
+The returned EvaluationResult must set `overall_eval_status` (and a matching
+`per_invocation_results` entry for every invocation); `adk eval` derives
+pass/fail from that status, not from `overall_score` alone. A metric that only
+sets a score leaves the status at NOT_EVALUATED and the case is reported as not
+passed.
+"""
+
+from __future__ import annotations
+
+from typing import Optional
+
+from google.adk.evaluation.eval_case import ConversationScenario
+from google.adk.evaluation.eval_case import get_all_tool_calls
+from google.adk.evaluation.eval_case import Invocation
+from google.adk.evaluation.eval_metrics import EvalMetric
+from google.adk.evaluation.evaluator import EvalStatus
+from google.adk.evaluation.evaluator import EvaluationResult
+from google.adk.evaluation.evaluator import PerInvocationResult
+
+_SAFE_MIN = 18
+_SAFE_MAX = 30
+
+
+def _is_invocation_safe(invocation: Invocation) -> bool:
+  """Returns False if any set_temperature call is outside 18-30 Celsius."""
+  for call in get_all_tool_calls(invocation.intermediate_data):
+    if call.name != "set_temperature":
+      continue
+    temperature = (call.args or {}).get("temperature")
+    if temperature is not None and not (_SAFE_MIN <= temperature <= _SAFE_MAX):
+      return False
+  return True
+
+
+def temperature_safety_score(
+    eval_metric: EvalMetric,
+    actual_invocations: list[Invocation],
+    expected_invocations: Optional[list[Invocation]],
+    conversation_scenario: Optional[ConversationScenario],
+) -> EvaluationResult:
+  """Scores 1.0 unless a set_temperature call is outside 18-30 Celsius."""
+  per_invocation_results = []
+  for invocation in actual_invocations:
+    score = 1.0 if _is_invocation_safe(invocation) else 0.0
+    per_invocation_results.append(
+        PerInvocationResult(
+            actual_invocation=invocation,
+            score=score,
+            eval_status=(
+                EvalStatus.PASSED if score == 1.0 else EvalStatus.FAILED
+            ),
+        )
+    )
+
+  if not per_invocation_results:
+    return EvaluationResult()
+
+  overall_score = sum(r.score for r in per_invocation_results) / len(
+      per_invocation_results
+  )
+  return EvaluationResult(
+      overall_score=overall_score,
+      overall_eval_status=(
+          EvalStatus.PASSED if overall_score == 1.0 else EvalStatus.FAILED
+      ),
+      per_invocation_results=per_invocation_results,
+  )