feat(testing): Add model matrix test harness and RFC

This adds the initial implementation and RFC for the model matrix test
harness. The harness enables running ADK conformance scenarios across
multiple backend/model combinations via a YAML-driven configuration to
produce structured pass/fail reports.

Included files:
- RFC-model-matrix-harness.md
- Unit tests and validators for the model matrix
- Temporary exploration scripts and test agents used during development

Author: GWeale

RFC-model-matrix-harness.md

@@ -0,0 +1,51 @@
+# RFC: Model Matrix Test Harness
+
+**Author:** George (gweale)  **Date:** 2026-02-25  **Status:** Draft
+
+## Summary
+
+This RFC proposes a YAML-driven model matrix test harness that runs ADK conformance scenarios across multiple backend/model combinations (Gemini API, Vertex AI) and produces structured pass/fail reports. No existing ADK source code is modified.
+
+## Motivation
+
+ADK supports multiple backends and models, but there is no automated way to validate that the full feature surface — tool calling, auth flows, streaming, session rewind, MCP integration, third-party adapters, and the public API contract — works consistently across each backend/model pair. Manual testing is slow and coverage gaps are invisible. A matrix harness would catch backend-specific regressions before they reach users and provide a repeatable conformance baseline for new model releases.
+
+## Proposal
+
+This RFC proposes a standalone test harness that lives entirely in the `tests/` tree. A YAML configuration file declares targets (backend + environment), models, and a list of scenario IDs to execute. An orchestrator loads this config into a validated Pydantic schema, expands it into row plans (one per target/model pair), and runs each row's scenarios against a subprocess ADK API server, collecting results into a structured JSON report with a Markdown summary.
+
+Proposed entry point:
+
+```python
+def run_matrix(
+    *,
+    config_path: Path,
+    output_dir: Optional[Path] = None,
+    environment: Optional[Mapping[str, str]] = None,
+    dry_run_override: Optional[bool] = None,
+    strict_mode_override: Optional[bool] = None,
+) -> MatrixReport
+```
+
+Key design elements:
+
+- **Config schema** — Pydantic models for targets, models, run options, and unit-test groups, with validation for required env vars and cross-references
+- **Extensible scenario registry** — ships with 25 scenarios across 9 domains (tool pipeline, MCP, auth, live/streaming, deployment parity, memory/rewind, third-party adapters, API contract, CLI flags). Adding a new scenario requires only writing a validator function, registering it as a `ScenarioDefinition`, and listing its ID in the YAML config — no orchestrator or schema changes needed
+- **Server runtime** — spins up a subprocess `adk api_server` per row, polls for readiness, and tears down after execution
+- **Live scenario runtime** — async client supporting `/run`, `/run_sse`, `/run_live` (WebSocket), concurrent prompt races, session rewind, and artifact operations
+- **Strict mode gate** — optionally enforces zero fail/error rows and flags unexpected skips
+- **Unit group integration** — runs named pytest selector groups as part of each row, reported alongside scenarios
+
+The architecture is designed so that the orchestrator, config schema, and reporting layer never need to change when new scenarios are added. Targets, models, and unit groups are also open for extension via the YAML config alone. Fully backward compatible — all code lives under `tests/`, no existing source files are modified.
+
+## Timeline
+
+N/A
+
+## Cross-workstream Impacts
+
+None — the harness consumes existing ADK public APIs and CLI commands without modifying them.
+
+## Outcome
+
+Success means any backend/model combination can be validated with a single `python run_matrix.py --config matrix.yaml` invocation, producing a structured report that CI or developers can use to catch regressions. Immediate next steps are enabling non-dry-run execution against live Gemini API and Vertex AI targets and integrating the harness into CI.

my_test_agent/__init__.py

@@ -0,0 +1 @@
+from . import agent

my_test_agent/agent.py

@@ -0,0 +1,7 @@
+from google.adk.agents import Agent
+
+root_agent = Agent(
+    name="test_agent",
+    model="gemini-3.5-flash",
+    instruction="You are a helpful assistant. Keep your response very short.",
+)

parse_html.py

@@ -0,0 +1,14 @@
+import sys
+from bs4 import BeautifulSoup
+
+class TextExtract(BeautifulSoup):
+    def get_text(self):
+        return super().get_text(separator='\n', strip=True)
+
+html = sys.stdin.read()
+soup = BeautifulSoup(html, 'html.parser')
+main_content = soup.find('article') or soup.find('main') or soup.body
+if main_content:
+    print(main_content.get_text(separator='\n', strip=True))
+else:
+    print("Could not find main content")

search_docs.py

@@ -0,0 +1,39 @@
+import urllib.request
+import json
+
+def search(query):
+    url = f"https://html.duckduckgo.com/html/?q={urllib.parse.quote(query)}"
+    req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
+    try:
+        html = urllib.request.urlopen(req).read().decode('utf-8')
+        from html.parser import HTMLParser
+        class MyParser(HTMLParser):
+            def __init__(self):
+                super().__init__()
+                self.in_a = False
+                self.results = []
+                self.current_link = ""
+            def handle_starttag(self, tag, attrs):
+                if tag == 'a':
+                    for attr in attrs:
+                        if attr[0] == 'href' and 'ad_provider' not in attr[1]:
+                            if attr[1].startswith('//duckduckgo.com/l/?uddg='):
+                                import urllib.parse
+                                self.current_link = urllib.parse.unquote(attr[1].split('uddg=')[1].split('&')[0])
+                                self.in_a = True
+            def handle_data(self, data):
+                if self.in_a and data.strip():
+                    self.results.append((data.strip(), self.current_link))
+            def handle_endtag(self, tag):
+                if tag == 'a':
+                    self.in_a = False
+
+        parser = MyParser()
+        parser.feed(html)
+        for text, link in parser.results[:10]:
+            print(f"{text}\n{link}\n")
+    except Exception as e:
+        print(f"Error: {e}")
+
+search("site:cloud.google.com/vertex-ai 'Interactions API'")
+search("site:ai.google.dev 'Interactions API'")

test_agent/agent.py

@@ -0,0 +1,8 @@
+from . import agent
+from google.adk.agents import Agent
+
+root_agent = Agent(
+    name="test_agent",
+    model="gemini-3.5-flash",
+    instruction="You are a helpful assistant. Please say 'Hello, I am gemini-3.5-flash'.",
+)

test_interactions_raw.py

@@ -0,0 +1,37 @@
+import asyncio
+import os
+import logging
+from google.genai import Client
+from google.genai import types
+
+async def check_models():
+    # Vertex AI Client
+    client_vertex = Client(
+        vertexai=True,
+        project="cloud-llm-preview1",
+        location="us-central1",
+        http_options={"api_version": "v1beta1"}
+    )
+    
+    models_to_test = [
+        "gemini-3-flash-preview",
+        "gemini-3.5-flash-preview"
+    ]
+    
+    for client, client_name in [(client_vertex, "Vertex AI")]:
+        print(f"\n=============================")
+        print(f"Testing Client: {client_name}")
+        print(f"=============================")
+        for model_id in models_to_test:
+            print(f"\n--- Testing model: {model_id} ---")
+            try:
+                interaction = await client.aio.interactions.create(
+                    model=model_id,
+                    input="Hello!"
+                )
+                print(f"Success!")
+            except Exception as e:
+                print(f"Error: {e}")
+
+if __name__ == "__main__":
+    asyncio.run(check_models())

test_model_info.py

@@ -0,0 +1,20 @@
+import asyncio
+from google.genai import Client
+
+async def check_models():
+    client = Client(
+        vertexai=True,
+        project="cloud-llm-preview1",
+        location="us-central1"
+    )
+    
+    print("Checking models...")
+    models = client.models.list()
+    for m in models:
+        if "gemini-3.5" in m.name or "gemini-2.5" in m.name:
+            print(f"Model: {m.name}")
+            print(f"  Supported Actions: {m.supported_actions if hasattr(m, 'supported_actions') else 'Unknown'}")
+            print(f"  Version: {m.version if hasattr(m, 'version') else 'Unknown'}")
+
+if __name__ == "__main__":
+    asyncio.run(check_models())

test_run.py

@@ -0,0 +1,21 @@
+import asyncio
+import os
+from my_test_agent.agent import root_agent
+from google.adk.runners import Runner
+from google.adk.sessions.in_memory_session_service import InMemorySessionService
+
+async def main():
+    runner = Runner(
+        app_name="test_app",
+        agent=root_agent,
+        session_service=InMemorySessionService(),
+        auto_create_session=True
+    )
+    print("Testing gemini-3.5-flash...")
+    events = await runner.run_debug("Hi, please confirm your model version.")
+    for e in events:
+        if e.author == 'test_agent' and e.content:
+            print("Response:", e.content.parts[0].text if e.content.parts else e.content)
+
+if __name__ == "__main__":
+    asyncio.run(main())

tests/unittests/model_matrix/__init__.py

@@ -0,0 +1,15 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import annotations