Skip to content

Commit fc3e021

Browse files
Squash PR 122 and PR123 and address reviewer feedback
This commit squashes PR 122 (Context Cache Hit Rate) and PR 123 (PerformanceEvaluator unification) rebased onto PR 121. - Added context cache hit rate metric to SystemEvaluator and CLI. - Unified one-sided and side-by-side performance metrics in PerformanceEvaluator. - Decoupled system and performance modules cleanly. - Preserved backward-compatible LLMAsJudge wrapper, restored deprecated SQL-batch path symbols in evaluators.py, and restored LLMAsJudge support in Client.evaluate(). - Added live integration test for PerformanceEvaluator LLM judge path. - Fixed notebook demos to use the new SystemEvaluator and PerformanceEvaluator. - Updated Client.evaluate docstring to reflect the new performance evaluation path. - Reformatted dashboard/app.py, tools.py, and improver_agent.py using pyink (no logic changes). - Verified notebook diffs are minimal format sync (cell IDs and JSON escaping, no execution outputs). TAG=agy CONV=7051de01-a124-4d4a-b178-81af860027bc
1 parent 0016418 commit fc3e021

107 files changed

Lines changed: 5088 additions & 5892 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -955,6 +955,18 @@ BQAA loop
955955
re-opens the choice for the session-aggregated `AI.GENERATE`
956956
tier with Option C (SQL / Python UDF) as the primary candidate.
957957
Unblocks the compile-harness PR.
958+
### Changed
959+
- **Renamed `CodeEvaluator` to `SystemEvaluator`** to align with its focus on system-level metrics. Kept `CodeEvaluator` as an alias for backward compatibility.
960+
- **Staged identical baseline copies and test renames** for lineage tracking.
961+
- **Unified One-Sided & Side-by-Side Performance Metrics** in the `PerformanceEvaluator`:
962+
- Purged obsolete criteria-list `LLMAsJudge` implementations, replacing them natively with `PerformanceEvaluator` for folded Tone, Faithfulness, Correctness, and Efficiency evaluations.
963+
- Decoupled system and performance modules cleanly, making `system_evaluator.py` pure to `SystemEvaluator`.
964+
- Overrode the backwards-compatible `LLMAsJudge` subclass in `evaluators.py` with required static factories for correctness, hallucination, and sentiment.
965+
- Removed `_JudgeCriterion` from public access in `system_evaluator.py` (now internal to `evaluators.py`).
966+
- **Migration Story**:
967+
- Users should no longer construct `_JudgeCriterion` objects directly. Instead, use `LLMAsJudge.add_criterion(name, prompt_template, score_key, threshold)` to add criteria to a judge instance.
968+
- `Client.evaluate(LLMAsJudge)` is no longer supported and will raise `TypeError`. Callers must migrate to using `PerformanceEvaluator` with the appropriate judge configurations.
969+
- The BigQuery-side `AI.GENERATE` batch SQL path has been removed in favor of Python-side per-session API calls. Callers depending on `render_ai_generate_judge_query` or `AI_GENERATE_JUDGE_BATCH_QUERY` template constants must migrate to `PerformanceEvaluator` which handles LLM evaluation programmatically. The legacy symbols are preserved as deprecated shims in `evaluators.py`.
958970

959971
## [0.2.3] - 2026-04-27
960972

README.md

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,9 @@ regressions — all through BigQuery SQL or Python.
2525
- Observability dashboards (SQL and BigFrames)
2626

2727
**Evaluation**
28-
- Code-based metrics (latency, turn count, error rate, token efficiency, cost)
29-
- LLM-as-Judge scoring (correctness, hallucination, sentiment)
30-
- Trajectory matching (exact, in-order, any-order)
31-
- Multi-trial evaluation with pass@k / pass^k
28+
- System metrics (latency, turn count, tool call error rate, token efficiency, time to first token, cost)
29+
- Performance Metrics (correctness, hallucination, sentiment, efficiency, etc)
30+
- Multi-trial system and performance metircs
3231
- Grader composition (weighted, binary, majority strategies)
3332
- Eval suite lifecycle management with graduation and saturation detection
3433
- Static quality validation (ambiguous tasks, class imbalance, suspicious thresholds)
@@ -123,10 +122,10 @@ src/bigquery_agent_analytics/
123122
│ └── formatter.py # Output formatting (json/text/table)
124123
125124
├── Evaluation
126-
│ ├── evaluators.py # SystemEvaluator + LLMAsJudge
127-
│ ├── trace_evaluator.py # Trajectory matching & replay
128-
│ ├── multi_trial.py # Multi-trial runner + pass@k
129-
── grader_pipeline.py # Grader composition pipeline
125+
│ ├── system_evaluator.py # SystemEvaluator
126+
│ ├── performance_evaluator.py # PerformanceEvaluator
127+
│ ├── multi_trial_performance_evaluator.py # MultiTrialPerformanceEvaluator
128+
── aggregate_grader.py # AggregateGrader
130129
│ ├── eval_suite.py # Eval suite lifecycle management
131130
│ └── eval_validator.py # Static validation checks
132131

0 commit comments

Comments
 (0)