wip: add eval optimization loop example

Author: xyxhhhhh

examples/optimization/eval_optimize_loop/README.md

@@ -0,0 +1,128 @@
+# Evaluation + Optimization Loop
+
+This example demonstrates a reproducible pipeline that connects evaluation,
+failure attribution, prompt optimization, validation regression checks, gate
+decisions, and audit reports.
+
+It is intentionally different from `ci_integration/`: CI integration focuses on
+PR gates and nightly write-back, while this example focuses on the complete
+review loop before deciding whether a candidate prompt is worth accepting.
+
+## Run
+
+The default mode is deterministic and does not require an API key:
+
+```bash
+cd examples/optimization/eval_optimize_loop
+PYTHONPATH=../../.. python run_pipeline.py --mode fake
+```
+
+If your environment only exposes `python3`, use:
+
+```bash
+PYTHONPATH=../../.. python3 run_pipeline.py --mode fake
+```
+
+To refresh the committed sample reports:
+
+```bash
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --update-sample-outputs
+```
+
+Trace replay mode also runs without model credentials. It first records fake
+outputs into generated `eval_mode: "trace"` evalsets, then evaluates those
+`actual_conversation` records without invoking `call_agent`:
+
+```bash
+PYTHONPATH=../../.. python run_pipeline.py --mode trace
+```
+
+The optional real optimizer path delegates candidate search to
+`AgentOptimizer.optimize` and requires the normal optimization dependencies and
+model environment variables:
+
+```bash
+export TRPC_AGENT_API_KEY="<your-key>"
+export TRPC_AGENT_BASE_URL="<your-endpoint>"
+export TRPC_AGENT_MODEL_NAME="<your-model>"
+PYTHONPATH=../../.. python run_pipeline.py --mode optimizer
+```
+
+## Inputs
+
+| File | Role |
+| --- | --- |
+| `train.evalset.json` | Training split used for baseline scoring and optimization feedback. |
+| `val.evalset.json` | Validation split used for regression and gate decisions. |
+| `optimizer.json` | Shared metric, optimizer, fake candidate, and gate configuration. |
+| `agent/prompts/system.md` | Baseline prompt source registered as `TargetPrompt("system_prompt")`. |
+
+The fake mode still uses `AgentEvaluator` for all train and validation scoring.
+Only candidate generation is deterministic, so the example can run in CI or on a
+laptop without model credentials.
+
+## Outputs
+
+Each run writes to `runs/<mode_timestamp>/`:
+
+| File or directory | Contents |
+| --- | --- |
+| `optimization_report.json` | Machine-readable baseline, candidate, delta, attribution, gate, and audit payload. |
+| `optimization_report.md` | Human-readable report for review. |
+| `candidate_prompts/` | Candidate prompt text per target field. |
+| `eval_*` | Raw `AgentEvaluator` result files for every phase. |
+| `config.snapshot.json` | Reproducible copy of the input config. |
+| `eval_metrics.snapshot.json` | Evaluator-compatible metric config extracted from `optimizer.json.evaluate`. |
+
+Committed examples live in `sample_outputs/`.
+
+## Case Coverage
+
+The six sample cases cover the required outcomes:
+
+| Case | Split | Expected behavior |
+| --- | --- | --- |
+| `train_refund_double_charge` | train | Candidate fixes a baseline failure. |
+| `train_password_reset` | train | Already passing; candidate is unchanged. |
+| `train_legacy_sync` | train | Candidate does not help. |
+| `val_vip_refund` | validation | Candidate creates a new pass. |
+| `val_plan_question` | validation | Candidate has no effect. |
+| `val_checkout_outage` | validation | Candidate regresses a critical case. |
+
+The default fake candidate improves train score but does not improve validation
+overall because it also introduces a new hard fail. The gate therefore rejects
+the candidate and records the overfitting reason.
+
+## Design Notes
+
+The pipeline treats evaluation results as the source of truth and converts
+`AgentEvaluator.get_executer(...).get_result()` into a stable per-case report.
+For each split, it records score, pass/fail status, normalized actual and
+expected final responses, metric details, and a short trace summary. Failure
+attribution is deterministic: invalid JSON or missing keys are classified as
+format violations; category mismatches indicate knowledge-recall gaps; priority
+mismatches are parameter errors; action mismatches are final-response errors.
+This keeps fake mode explainable and also gives real optimizer runs a fallback
+when no LLM judge reason is available.
+
+Candidate prompts are validated on the held-out validation set before any source
+prompt is updated. The configurable gate checks minimum validation improvement,
+new hard failures, critical-case regressions, and cost budget. It also includes
+an explicit overfitting guard: when training improves but validation does not,
+the candidate is rejected even if some individual validation case improves. All
+decisions include concrete reasons, so a reviewer can see whether rejection came
+from score, hard-fail, critical-case, cost, or overfitting policy.
+
+Auditability is handled by snapshotting config, writing candidate prompt files,
+preserving raw evaluator outputs, and emitting both JSON and Markdown reports.
+The JSON report is suitable for CI parsing, while the Markdown report summarizes
+score deltas, per-case outcomes, failure clusters, and the final accept/reject
+decision for humans. Fake mode makes this loop reproducible without API keys;
+real mode can swap in `AgentOptimizer` while keeping the same validation, gate,
+and report machinery. The source prompt is never overwritten by this example;
+reviewers can inspect `candidate_prompts/` and decide whether to copy or submit
+the change through a normal PR. Hidden samples can reuse the same gate because
+all decisions are based on held-out validation deltas rather than train-set
+scores alone. Trace-style replay can also be added by replacing `call_agent`
+with eval cases that provide `actual_conversation`, while the attribution, delta,
+gate, and report stages remain unchanged.

examples/optimization/eval_optimize_loop/agent/__init__.py

@@ -0,0 +1,2 @@
+"""Agent entry points for the eval-optimize-loop example."""
+

examples/optimization/eval_optimize_loop/agent/agent.py

@@ -0,0 +1,132 @@
+# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
+#
+# Copyright (C) 2026 Tencent. All rights reserved.
+#
+# tRPC-Agent-Python is licensed under Apache-2.0.
+"""Deterministic fake agent for the eval-optimize-loop example.
+
+The example's default mode must run without an API key, so this module exposes
+an async ``call_agent`` that derives behavior from the current prompt text on
+disk. The pipeline still uses AgentEvaluator for scoring; this module only
+stands in for the model.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+
+
+SYSTEM_PROMPT_PATH = Path(__file__).parent / "prompts" / "system.md"
+
+_SPACE_RE = re.compile(r"\s+")
+
+
+def _compact_json(payload: dict[str, str]) -> str:
+    return json.dumps(payload, sort_keys=True, separators=(",", ":"))
+
+
+def normalize_json_text(raw: str) -> str:
+    """Normalize JSON-like model output for stable exact matching."""
+    try:
+        parsed = json.loads(raw)
+    except json.JSONDecodeError:
+        return _SPACE_RE.sub(" ", raw.strip())
+    return _compact_json(parsed)
+
+
+def _prompt_flags() -> dict[str, bool]:
+    prompt = SYSTEM_PROMPT_PATH.read_text(encoding="utf-8").lower()
+    return {
+        "refund_rule": "treat double charge" in prompt or "vip refund requests" in prompt,
+        "keep_outage_p1": "p1 for urgent production outages" in prompt,
+        "keep_plan_p3": "p3 for low-risk informational requests" in prompt,
+        "overfit_payment_outage": "overfit_payment_outage" in prompt,
+    }
+
+
+def answer_for_query(query: str) -> str:
+    """Return the fake model answer for one support ticket."""
+    text = query.lower()
+    flags = _prompt_flags()
+
+    if "production checkout outage" in text:
+        if flags["overfit_payment_outage"]:
+            return _compact_json({
+                "category": "billing",
+                "priority": "p2",
+                "action": "refund_review",
+            })
+        priority = "p1" if flags["keep_outage_p1"] else "p2"
+        return _compact_json({
+            "category": "technical",
+            "priority": priority,
+            "action": "escalate",
+        })
+
+    if "vip customer" in text and "double charged" in text:
+        if flags["refund_rule"]:
+            return _compact_json({
+                "category": "billing",
+                "priority": "p1",
+                "action": "refund_review",
+            })
+        return _compact_json({
+            "category": "account",
+            "priority": "p2",
+            "action": "answer",
+        })
+
+    if "double charged" in text or "refund" in text:
+        if flags["refund_rule"]:
+            return _compact_json({
+                "category": "billing",
+                "priority": "p2",
+                "action": "refund_review",
+            })
+        return _compact_json({
+            "category": "account",
+            "priority": "p2",
+            "action": "answer",
+        })
+
+    if "password reset" in text or "email address" in text:
+        return _compact_json({
+            "category": "account",
+            "priority": "p3",
+            "action": "answer",
+        })
+
+    if "plan comparison" in text or "pricing tiers" in text:
+        priority = "p3" if flags["keep_plan_p3"] else "p2"
+        return _compact_json({
+            "category": "billing",
+            "priority": priority,
+            "action": "answer",
+        })
+
+    if "mobile app crashes" in text:
+        return _compact_json({
+            "category": "technical",
+            "priority": "p2",
+            "action": "troubleshooting",
+        })
+
+    if "legacy desktop sync" in text:
+        return _compact_json({
+            "category": "technical",
+            "priority": "p2",
+            "action": "troubleshooting",
+        })
+
+    return _compact_json({
+        "category": "technical",
+        "priority": "p2",
+        "action": "troubleshooting",
+    })
+
+
+async def call_agent(query: str) -> str:
+    """AgentEvaluator / AgentOptimizer compatible black-box entry point."""
+    return answer_for_query(query)

examples/optimization/eval_optimize_loop/agent/prompts/system.md

@@ -0,0 +1,23 @@
+You are a support triage assistant.
+
+Return one compact JSON object with exactly these keys:
+- category
+- priority
+- action
+
+Known categories:
+- account
+- billing
+- technical
+
+Known priorities:
+- p1 for urgent production outages
+- p2 for normal user-impacting issues
+- p3 for low-risk informational requests
+
+Known actions:
+- escalate
+- refund_review
+- troubleshooting
+- answer
+

examples/optimization/eval_optimize_loop/optimizer.json

@@ -0,0 +1,60 @@
+{
+  "evaluate": {
+    "metrics": [
+      {
+        "metric_name": "final_response_avg_score",
+        "threshold": 1.0,
+        "criterion": {
+          "final_response": {
+            "text": {
+              "match": "exact",
+              "case_insensitive": false
+            }
+          }
+        }
+      }
+    ],
+    "num_runs": 1
+  },
+  "gate": {
+    "min_validation_score_delta": 0.1,
+    "allow_new_hard_fail": false,
+    "critical_case_ids": ["val_vip_refund", "val_checkout_outage"],
+    "max_cost_usd": 0.01
+  },
+  "fake_optimizer": {
+    "candidate_id": "candidate_refund_rule_overfit",
+    "seed": 42,
+    "notes": "Adds refund handling learned from train failures but intentionally omits the outage guard to demonstrate overfitting rejection."
+  },
+  "optimize": {
+    "eval_case_parallelism": 1,
+    "stop": {
+      "required_metrics": "all"
+    },
+    "algorithm": {
+      "name": "gepa_reflective",
+      "seed": 42,
+      "reflection_lm": {
+        "model_name": "${TRPC_AGENT_MODEL_NAME}",
+        "base_url": "${TRPC_AGENT_BASE_URL}",
+        "api_key": "${TRPC_AGENT_API_KEY}",
+        "generation_config": {
+          "max_tokens": 2048,
+          "temperature": 0.4
+        }
+      },
+      "candidate_selection_strategy": "pareto",
+      "module_selector": "round_robin",
+      "frontier_type": "instance",
+      "reflection_minibatch_size": 3,
+      "reflection_history_top_k": 2,
+      "skip_perfect_score": false,
+      "use_merge": false,
+      "max_metric_calls": 18,
+      "score_threshold": 1.0,
+      "max_iterations_without_improvement": 2
+    }
+  }
+}
+