feat: Add ADK evaluation samples: shared home-automation agent

Start contributing/samples/evaluation/, a family of single-concept samples that
demonstrate ADK evaluation through the `adk eval` CLI over one shared agent.

This first change adds the foundation shared by every sub-sample:

- home_automation_agent/: the shared agent (device-control tools) that all
  sub-samples evaluate.
- README.md: an overview of the family and how each sub-sample is run.

Subsequent changes each add one single-concept sub-sample folder.

Co-authored-by: Haran Rajkumar <haranrk@google.com>
PiperOrigin-RevId: 952558506

Author: haranrk

contributing/samples/evaluation/README.md

@@ -0,0 +1,58 @@
+# ADK evaluation samples
+
+## Overview
+
+A family of single-concept samples that each show one way to evaluate the *same*
+shared home-automation agent with the `adk eval` CLI. Every sample points `adk eval` at one agent and differs only in its eval data and criteria, so you can
+compare evaluation techniques (deterministic reference matching, custom metrics,
+LLM-as-a-judge, rubrics, and user simulation) side by side.
+
+## The shared agent
+
+`home_automation_agent/` is a small agent that controls smart-home devices and
+temperatures. Its five tools (`get_device_info`, `set_device_info`,
+`get_temperature`, `set_temperature`, `list_devices`) are deterministic, backed by
+in-memory state, so eval trajectories are reproducible. The module exposes
+`reset_data()`, which `adk eval` calls to reset that state between eval cases.
+
+Every sample evaluates this same agent. `adk eval` takes the agent path and the
+eval-set path as two separate arguments, so each sub-sample folder holds only
+eval data and its criteria config, never a copy of the agent code.
+
+## How evaluation runs
+
+`adk eval` runs in two phases: (1) live inference, where it actually runs the
+agent against each eval input to produce responses and tool calls, and then (2)
+scoring, where it compares that output against the case's criteria. Because phase 1
+runs the real agent, a model credential is required for every sample, even the
+deterministic ones. Provide a Gemini API key in `home_automation_agent/.env`,
+or configure Vertex.
+Samples that use an LLM judge or a user simulator make additional model calls, but
+they resolve through the same model registry and credentials.
+
+Because live responses vary from run to run, the deterministic, reference-based
+criteria use lenient response thresholds (e.g. `response_match_score` at
+`0.5`) so that harmless phrasing differences don't fail an otherwise-correct
+answer.
+
+## Samples
+
+| Sample | Concept | Criteria |
+| ------ | ------- | -------- |
+
+## Graph
+
+```mermaid
+graph TD
+    A[home_automation_agent] --> B(get_device_info)
+    A --> C(set_device_info)
+    A --> D(get_temperature)
+    A --> E(set_temperature)
+    A --> F(list_devices)
+```
+
+## Related Guides
+
+- Evaluation overview: https://adk.dev/evaluate/
+- Evaluation criteria reference: https://adk.dev/evaluate/criteria/
+- User simulation guide: https://adk.dev/evaluate/user-sim/

contributing/samples/evaluation/home_automation_agent/__init__.py

@@ -0,0 +1,15 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from . import agent

contributing/samples/evaluation/home_automation_agent/agent.py

@@ -0,0 +1,103 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""A small home-automation agent shared by the evaluation samples.
+
+All tools are deterministic (backed by in-memory dicts) so that eval
+trajectories are reproducible. `reset_data()` is picked up by `adk eval` to
+reset state between eval cases.
+"""
+
+from google.adk import Agent
+
+_DEVICES = None
+_TEMPERATURES = None
+
+
+def reset_data() -> None:
+  """Resets in-memory state. Called by adk eval between eval cases."""
+  global _DEVICES, _TEMPERATURES
+  _DEVICES = {
+      "device_1": {"status": "ON", "location": "Living Room"},
+      "device_2": {"status": "OFF", "location": "Bedroom"},
+      "device_3": {"status": "OFF", "location": "Kitchen"},
+  }
+  _TEMPERATURES = {"Living Room": 22, "Bedroom": 20, "Kitchen": 24}
+
+
+# Initialize module-level state at import time.
+reset_data()
+
+
+def get_device_info(device_id: str) -> dict:
+  """Returns the status and location of a device, or an error string."""
+  return _DEVICES.get(device_id, {"error": "Device not found"})
+
+
+def set_device_info(device_id: str, status: str) -> str:
+  """Sets a device status to 'ON' or 'OFF'."""
+  if device_id not in _DEVICES:
+    return "Device not found"
+  _DEVICES[device_id]["status"] = status
+  return f"Device {device_id} is now {status}."
+
+
+def get_temperature(location: str) -> str:
+  """Returns the current temperature (Celsius) of a location."""
+  if location not in _TEMPERATURES:
+    return "Location not found"
+  return f"{_TEMPERATURES[location]}"
+
+
+def set_temperature(location: str, temperature: int) -> str:
+  """Sets the target temperature (Celsius) for a location.
+
+  Acceptable range is 18-30 Celsius. Do not call this tool with a value
+  outside that range.
+  """
+  if location not in _TEMPERATURES:
+    return "Location not found"
+  _TEMPERATURES[location] = temperature
+  return f"Temperature in {location} set to {temperature}C."
+
+
+def list_devices(status: str = "", location: str = "") -> list:
+  """Lists devices, optionally filtered by status and/or location."""
+  result = []
+  for device_id, info in _DEVICES.items():
+    if (not status or info["status"] == status) and (
+        not location or info["location"] == location
+    ):
+      result.append({"device_id": device_id, **info})
+  return result
+
+
+root_agent = Agent(
+    name="home_automation_agent",
+    description="Controls smart-home devices and temperature.",
+    instruction=(
+        "You are a home-automation assistant. Use the available tools to"
+        " inspect and control devices and temperatures. When the user asks to"
+        " change something, call the matching tool, then confirm the result in"
+        " one short sentence. If the user asks to set a temperature outside the"
+        " safe range of 18-30 Celsius, refuse and do not call the tool."
+    ),
+    tools=[
+        get_device_info,
+        set_device_info,
+        get_temperature,
+        set_temperature,
+        list_devices,
+    ],
+)