feat(scripts): add latency report and scope-aware quality eval (#156)

* Add latency report, scope-aware quality eval, and sample reports

- Rename trace_latency to latency_report for consistency with quality_report
- Add --report and --output-json flags to latency_report
- Add --session flag to quality_report for single-session evaluation
- Add --config flag for scope-aware eval with declined category
- Add sample_quality_report.md and sample_latency_report.md
- Update README with scope-aware eval docs and latency report section
- Add .env and example run artifacts to .gitignore

* Improve multi-turn support, A2A null handling, and scope evaluation

- get_user_input: return last user message (not first) for multi-turn alignment
- get_a2a_response: handle null/empty A2A content as "(no response)" instead of
  falling through to earlier turns, preventing question/response mismatches
- Always include declined category in response_usefulness metrics — lets the
  LLM judge correctly classify legitimate scope declines even without --config
- _load_agent_config: implement auto-discovery of eval/data/agent_context.json
  (the docstring already described this behavior but the code returned None)
- Add Declined Sessions section to markdown report
- Add 14 new tests covering get_user_input and get_a2a_response

* Address PR #156 review: bug fixes, dead code removal, tests, logging

quality_report.py:
- B2: Replace stale config cache with dict keyed by config_path
- B3: Raise FileNotFoundError instead of sys.exit for missing config
- M3: Consistent get_a2a_response return shape (always tuple)
- M4: Comment explaining is_a2a=True for "(no response)" sessions
- M7: Remove duplicate import json as _json
- M9: Document that 'reason' field in --config is documentation-only
- L7: Store samples as None (not string) when unset
- Fail fast with clear error when --session-ids-file has no valid IDs

latency_report.py:
- H1: Remove dead parse_period() function and unused import re
- H4: Add _load_dotenv(), _configure_logging(), --env CLI flag
- M6: Document that stitch_a2a_traces mutates traces in place
- M10: Extract A2A noise events to module-level frozenset
- L2: Move Counter import and _script_dir to module level
- L3: Replace print() with logger.* for operational messages
- Second review: --output-json - sends human output to stderr

tests:
- H2: Add test_latency_report_helpers.py (26 tests for pure helpers)
- H3: Add TestBuildScopeContext (6 tests) and TestLoadAgentConfig (4 tests)
- M1: Remove unused import pytest from top level
- Update test_invalid_json_string for M3 consistent return shape

Formatting: pyink + isort applied via autoformat.sh

* Address second review: autoformat scripts/, new findings, polish

B1: Add scripts/ to autoformat.sh scope; run pyink+isort on both
    scripts (now consistent 2-space indent throughout)

N1: Guard --sdk-tree + --output-json - (SDK trace.render() writes
    directly to stdout, bypassing the stderr routing)

N4: Add test_cache_isolates_paths — proves two different config paths
    return two different configs (regression guard for B2 fix)

N5: Include session ID in A2A JSON parse warning for debuggability

M2: Add docstring to get_eval_metrics documenting returned metrics
    and the always-present declined category

M8: Qualify run_eval.py path in --session-ids-file help text

N3: Add --env example to README direct-Python usage section

L4: Complete latency_report.py module docstring with all flags
    (--no-waterfall, --sdk-tree, --report, --output-json, --env)

79 tests pass, autoformat clean (pyink + isort).

* Address remaining review items: rebase, polish, cleanup

L1:  Remove GFM trailing whitespace from sample markdown files
L6:  Add comment about approximate alignment in waterfall markers
L8:  Clarify --time-period help text in latency_report.py
L9:  Scope .gitignore *.log to scripts/ and examples/ only
L10: Remove duplicate env validation from latency_report.sh
     (Python script now handles validation with structured logging)
L11: Rebased onto origin/main (17 commits ahead, 0 behind)

79 tests pass, autoformat clean.

* Regenerate sample reports from live data

Ran both scripts against live BigQuery data to verify end-to-end:
- quality_report.py --limit 20 --report --samples 3
- latency_report.py --limit 3 --report

Sanitized project-specific details for the sample files.
Stripped GFM trailing whitespace (L1).

* Align --time-period default between quality and latency scripts

Change latency_report.py --time-period default from None to "all" to
match quality_report.py. Convert "all" back to None before building
the TraceFilter so runtime behavior is unchanged.

Also regenerate sample_latency_report.md from live data and add .adk/
to .gitignore.

Author: evekhm

.gitignore

@@ -11,11 +11,20 @@ build/
 .venv/
 venv/
 env/
+.adk/
 uv.lock
+.env
 
 # Script outputs
 scripts/reports/
 
+# Example run artifacts
+examples/*/reports/
+examples/*/reports_*/
+examples/*/trials_*/
+scripts/**/*.log
+examples/**/*.log
+
 # Local workspace metadata
 .code*/
 deploy/streaming_evaluation/.streaming_evaluation_state.json

autoformat.sh

@@ -21,9 +21,9 @@ set -euo pipefail
 REPO_ROOT="$(cd "$(dirname "$0")" && pwd)"
 
 echo "==> Running isort (import sorting)..."
-isort "${REPO_ROOT}/src/" "${REPO_ROOT}/tests/" "${REPO_ROOT}/examples/"
+isort "${REPO_ROOT}/src/" "${REPO_ROOT}/tests/" "${REPO_ROOT}/examples/" "${REPO_ROOT}/scripts/"
 
 echo "==> Running pyink (code formatting)..."
-pyink --config "${REPO_ROOT}/pyproject.toml" "${REPO_ROOT}/src/" "${REPO_ROOT}/tests/" "${REPO_ROOT}/examples/"
+pyink --config "${REPO_ROOT}/pyproject.toml" "${REPO_ROOT}/src/" "${REPO_ROOT}/tests/" "${REPO_ROOT}/examples/" "${REPO_ROOT}/scripts/"
 
 echo "==> Done. All Python files formatted."

scripts/README.md

@@ -2,6 +2,11 @@
 
 Standalone scripts for the BigQuery Agent Analytics SDK.
 
+| Script | Description |
+|--------|-------------|
+| [quality_report](#quality-report) | LLM-as-a-judge evaluation over agent sessions |
+| [latency_report](#latency-report-1) | Timing tree and waterfall for agent traces with A2A stitching |
+
 ## Quality Report
 
 Runs LLM-as-a-judge evaluation over agent sessions stored in BigQuery
@@ -56,6 +61,7 @@ EVAL_MODEL_ID=gemini-2.5-flash
 ./scripts/quality_report.sh --session-ids-file ids.json  # evaluate specific sessions
 ./scripts/quality_report.sh --output-json report.json    # write structured JSON output
 ./scripts/quality_report.sh --threshold 15          # unhelpful rate warning at 15%
+./scripts/quality_report.sh --config config.json    # scope-aware eval with config
 ```
 
 Or run the Python script directly:
@@ -98,11 +104,58 @@ These filters can be combined (e.g. `--app-name my_agent --session-ids-file ids.
 The evaluation uses two categorical metrics:
 
 - **response_usefulness** - Whether the agent's response provides a genuinely
-  useful answer. Categories: `meaningful`, `unhelpful`, `partial`.
+  useful answer. Categories: `meaningful`, `declined`, `unhelpful`, `partial`.
 
 - **task_grounding** - Whether the response is grounded in tool-retrieved data
   or fabricated. Categories: `grounded`, `ungrounded`, `no_tool_needed`.
 
+The **`declined`** category is always available — the LLM judge can classify
+polite refusals of out-of-scope questions as correct behavior rather than
+marking them as `unhelpful`.
+
+### Scope-Aware Evaluation (`--config`)
+
+For more accurate scope evaluation, provide a config file that tells the
+LLM judge exactly which topics your agent intentionally does not handle:
+
+```bash
+./scripts/quality_report.sh --config agent_context.json --report
+```
+
+The script also auto-discovers `eval/data/agent_context.json` relative to
+the repo root or script directory, so `--config` is only needed to point
+at a non-default location.
+
+Create a JSON config file with `scope_decisions`:
+
+```json
+{
+  "scope_decisions": [
+    {
+      "topic": "stock_options",
+      "decision": "out_of_scope",
+      "reason": "No tool or data source covers equity compensation"
+    },
+    {
+      "topic": "salary_bands",
+      "decision": "out_of_scope",
+      "reason": "Confidential compensation data"
+    },
+    {
+      "topic": "promotions",
+      "decision": "out_of_scope",
+      "reason": "No tool covers career progression"
+    }
+  ]
+}
+```
+
+Without a config, the LLM judge can still classify obvious declines as
+`declined`, but it won't know which specific topics are out of scope. With
+the config, the judge is told exactly which topics are out of scope, so it
+can correctly classify polite refusals as `declined` (correct behavior)
+rather than `unhelpful` (a bug).
+
 ### A2A Support
 
 The script automatically detects and resolves responses from remote A2A
@@ -111,4 +164,65 @@ The script automatically detects and resolves responses from remote A2A
 
 ### Sample report output
 
-[Sample report output](sample_report.md)
+[Sample quality report](sample_quality_report.md)
+
+---
+
+## Latency Report
+
+Fetches agent traces from BigQuery and renders a hierarchical timing tree
+with per-span latency and a waterfall timeline. Automatically stitches
+A2A (Agent-to-Agent) remote sessions to show full cross-agent latency
+breakdown — including LLM call times inside remote agents that would
+otherwise appear as a black box.
+
+### Usage
+
+```bash
+./scripts/latency_report.sh                              # latest trace
+./scripts/latency_report.sh --limit 5                    # last 5 traces with summary
+./scripts/latency_report.sh --time-period 1h             # traces from the last hour
+./scripts/latency_report.sh --session <session_id>       # specific session
+./scripts/latency_report.sh --app-name my_agent          # filter by root agent name
+./scripts/latency_report.sh --verbose                    # show questions and responses
+./scripts/latency_report.sh --no-stitch                  # skip A2A session stitching
+./scripts/latency_report.sh --env path/to/.env           # use a specific .env file
+```
+
+Or run the Python script directly:
+
+```bash
+python scripts/latency_report.py --limit 5 --time-period 1h
+python scripts/latency_report.py --env path/to/.env --limit 5
+```
+
+### Output
+
+The script produces three views for each trace:
+
+1. **Timing tree** — hierarchical span view with latency annotations,
+   tool names, and A2A boundary markers
+2. **Waterfall chart** — ASCII bar chart showing time distribution
+3. **SDK trace tree** — the SDK's built-in `trace.render()` output
+
+When multiple traces are fetched (`--limit > 1`), a **summary table**
+shows aggregate latency statistics (avg, P50, P95, min, max) and
+per-agent breakdown.
+
+### A2A Session Stitching
+
+When a supervisor agent calls a remote agent via A2A, the parent trace
+only records `AGENT_STARTING` and `AGENT_COMPLETED` for the remote
+agent — the internal LLM and tool spans are logged in a separate
+BigQuery session.
+
+The script automatically:
+1. Detects `A2A_INTERACTION` events in the parent trace
+2. Extracts the remote session ID from `content.metadata.adk_session_id`
+3. Fetches the remote agent's spans and inlines them as children
+
+Use `--no-stitch` to disable this behavior.
+
+### Sample report output
+
+[Sample latency report](sample_latency_report.md)