feat(optimization): add reproducible eval optimize loop

Author: Adonis-a233

examples/optimization/eval_optimize_loop/.gitignore

@@ -2,3 +2,4 @@
 # runs/latest prompt snapshots are kept in VCS as example deliverables.
 __pycache__/
 _sdk_eval_metrics.json
+runs/latest/agent_optimizer/

examples/optimization/eval_optimize_loop/README.md

@@ -1,71 +1,110 @@
-# eval_optimize_loop
+# Evaluation + Optimization Loop
 
-> Part of the `examples/optimization` series. Where [`quickstart`](../quickstart)
-> drives a live `AgentOptimizer` (GEPA) against a real agent, this example focuses
-> on the **closed loop around** optimization — baseline evaluation, failure
-> attribution, candidate validation, an acceptance gate, and an auditable report —
-> and runs fully reproducibly in fake/trace mode **without an API key**.
+## 1. Purpose
 
-This example implements a reproducible Evaluation + Optimization closed loop:
+This example implements the issue requirement for a reproducible Evaluation + Optimization pipeline. It is not only an `AgentOptimizer` quickstart: it wraps optimization with baseline evaluation, failure attribution, validation regression, gate decisions, and audit artifacts.
+
+The default `fake` mode runs without model credentials. The `live` mode uses a real `LlmAgent` bridge and invokes `AgentOptimizer.optimize` against a `TargetPrompt`.
+
+## 2. Pipeline Stages
+
+The pipeline runs six stages:
+
+1. Baseline evaluation: score train and validation sets separately, including metric scores, pass/fail, reasons, and key trace fields.
+2. Failure attribution: cluster failures into `final_response_mismatch`, `tool_call_error`, `parameter_error`, `llm_rubric_not_met`, `knowledge_recall_insufficient`, and `format_error`.
+3. Optimization execution: fake mode applies a deterministic candidate; live mode calls `AgentOptimizer.optimize` with `TargetPrompt.add_path("system_prompt", ...)`.
+4. Candidate validation: rerun train and validation sets and compute per-case deltas such as `new_pass`, `new_fail`, `score_up`, and `score_down`.
+5. Acceptance gate: require validation gain, no new hard fail, no key-case regression, no train-up/validation-down overfit, and cost within budget.
+6. Audit persistence: write prompt snapshots, scores, deltas, gate reasons, cost, duration, seed, and config snapshots.
+
+## 3. Directory Layout
 
 ```text
-baseline evaluation
-  -> failure attribution
-  -> prompt candidate generation
-  -> validation regression
-  -> acceptance gate
-  -> auditable JSON/Markdown report
+examples/optimization/eval_optimize_loop/
+├── agent/
+│   ├── __init__.py
+│   └── agent.py
+├── prompts/
+│   └── system.md
+├── train.evalset.json
+├── val.evalset.json
+├── case_meta.json
+├── optimizer.json
+├── optimizer.sdk.json
+├── run.py
+├── optimization_report.json
+└── optimization_report.md
 ```
 
-Run without API keys:
+## 4. Inputs
+
+- `train.evalset.json`: training evaluation set.
+- `val.evalset.json`: validation evaluation set; it must be a different file from train.
+- `optimizer.json`: outer-loop configuration for mode, metrics, fake candidate patch, and gate thresholds.
+- `prompts/system.md`: baseline prompt source registered as the optimization target.
+- `case_meta.json`: out-of-schema metadata for key cases, rubric kinds, and attribution hints.
+- `optimizer.sdk.json`: live-only SDK optimizer config passed to `AgentOptimizer.optimize`.
+
+## 5. Outputs
+
+- `optimization_report.json`: machine-readable audit report with baseline, candidate, delta, gate, attribution, optimizer status, cost, duration, seed, and config snapshot.
+- `optimization_report.md`: human-readable decision summary.
+- `runs/latest/baseline_prompt.md`: exact baseline prompt snapshot.
+- `runs/latest/candidate_prompt.md`: candidate prompt snapshot.
+- `runs/latest/agent_optimizer/`: live-only raw SDK artifacts, including `RoundRecord`-backed round files, `result.json`, `summary.txt`, and `best_prompts/`.
+
+## 6. Run Modes
+
+Fake mode:
 
 ```bash
-# First run may spend time on a one-off `uv sync`; the loop itself is ~seconds.
-uv run python examples/optimization/eval_optimize_loop/run.py
+python examples/optimization/eval_optimize_loop/run.py --mode fake
 ```
 
-Set `YUN_LOG_LEVEL=DEBUG` for more verbose logs (default `INFO`).
+Live mode:
 
-Inputs:
+```bash
+set TRPC_AGENT_API_KEY=...
+set TRPC_AGENT_BASE_URL=...
+set TRPC_AGENT_MODEL_NAME=...
+python examples/optimization/eval_optimize_loop/run.py --mode live
+```
+
+`fake` mode uses a deterministic fake model, fake judge, and scripted candidate so the full loop runs without API keys. `live` mode uses `agent/agent.py`, creates a fresh `LlmAgent` for each call, and invokes `AgentOptimizer.optimize`.
+
+## 7. Customizing The Agent
+
+Edit `agent/agent.py` when connecting a real business agent.
+
+Key constraints:
 
-- `prompts/baseline_system.md` — target prompt being optimized.
-- `train.evalset.json` / `val.evalset.json` — SDK-clean evalsets (trace mode).
-- `case_meta.json` — per-case `key` / `rubric` / `tool_intent` (kept out of the
-  evalset so `EvalSet` stays schema-clean).
-- `optimizer.json` — metric weights, scripted candidate patch, and gate thresholds.
+- `make_call_agent(prompt_path)` must return an async function with the exact optimizer contract `async (query: str) -> str`.
+- `create_agent(prompt_path)` must re-read the prompt file every time so candidates written by `AgentOptimizer` take effect immediately.
+- `TargetPrompt.add_path("system_prompt", path)` must point to the same prompt file that the agent actually reads.
+- For HTTP, CLI, remote config, or multi-agent pipelines, keep the outer contract the same and replace only the bridge implementation.
 
-Outputs:
+The outer report still computes richer trace-style scoring. The SDK optimizer itself receives final-text responses through `call_agent`, so `optimizer.sdk.json` intentionally avoids metrics that require full session traces.
 
-- `optimization_report.json` / `optimization_report.md`
-- `runs/latest/baseline_prompt.md` / `runs/latest/candidate_prompt.md`
+## 8. Design And Validation
 
-The sample has 6 cases:
+Failure attribution is rule-based over structured signals, not case ids. Each case records final response, tool trajectory, rubric sub-scores, and expected/actual tool calls. Rubric failures map to `format_error` or `llm_rubric_not_met`; tool mismatches map to tool, parameter, spurious-call, or knowledge-recall categories.
 
-- train: two optimizable failures and one optimization-ineffective format case.
-- validation: one new pass, one hard regression, and one soft degradation.
+The gate is validation-first. A candidate is accepted only if validation mean improves by the configured threshold, no new hard fail appears, key validation cases do not regress, train improvement does not coincide with validation loss, and cost is within budget.
 
-## 设计说明（四支柱）
+The bundled fake candidate intentionally improves two train cases and one validation case while damaging two key validation cases. The expected sample decision is `REJECT`, demonstrating overfit rejection.
 
-**失败归因（阶段 2）。** 归因完全基于结构化评测信号，不依赖 case 命名。每条 case
-记录三项 metric 子分（final_response / tool_trajectory / rubric）与关键轨迹（query、
-expected/actual 工具与回复）。`classify_tool_failure` 据「期望轨迹 vs 实际轨迹」判类：
-期望调用权威检索工具却没调用 → `knowledge_recall_insufficient`；调全了期望工具又多调
-→ `spurious_tool_call`；首工具名对但参数不同 → `parameter_error`；否则 `tool_call_error`。
-rubric 维度由 `case_meta.json` 显式声明（`json_format`/`no_tool`/`single_tool`），失败时
-映射为 `format_error` 或 `llm_rubric_not_met`。归因只统计 baseline 失败。
+Verified fake command:
 
-**接受策略（阶段 5）。** Gate 以验证集为先，五项可配置约束全过才 ACCEPT：① 验证集均分
-提升 ≥ `min_val_score_gain`；② 无新增 hard fail；③ 无「关键 case」退化（关键性由
-`case_meta.json` 的 `key=true` 标记，而非把所有验证 case 一概视为关键）；④ 非过拟合
-（训练涨而验证跌）；⑤ 优化成本 ≤ `max_cost_usd`。各检查相互独立，便于定位拒绝原因。
+```bash
+C:\Users\27303\PycharmProjects\Yun\.venv\Scripts\python.exe examples\optimization\eval_optimize_loop\run.py --mode fake
+```
+
+Observed sample result:
 
-**防过拟合。** 第 ④ 项专门拦截「训练大幅提升、验证退化」：本例候选给 baseline 注入
-激进检索行为，训练集 +0.53 但验证集回落，gate 据此 REJECT。关键 case 退化（③）与新增
-hard fail（②）提供正交的二次保险，即便总分变化很小也能拦住有害候选。
+```text
+train: 0.25 -> 0.7833
+validation: 0.7333 -> 0.6667
+decision: REJECT
+```
 
-**产物审计（阶段 6）。** `optimization_report.json` 持久化 baseline/candidate 逐 case 分数与
-轨迹、逐 case delta、失败归因、gate 各检查、决策理由、成本/耗时/seed、prompt 的 SHA-256 与
-config 快照；`runs/latest/` 留存 baseline 与候选 prompt 全文。`.md` 顶部由 `narrate_report`
-依据 gate/delta 数据生成「人话总结」，确定性、无需模型，换输入也不会失真。SDK 桥接通过
-`EvalSet.model_validate_json` 校验评测集，并用 trace-only `AgentEvaluator` 跑一次冒烟，证明
-管线确实接到真实 SDK 评测器；fake/trace 模式仅在评分/优化处用确定性替身以保证无 key 可复现。
+Known limits: live mode requires SDK dependencies plus `TRPC_AGENT_API_KEY`, `TRPC_AGENT_BASE_URL`, and `TRPC_AGENT_MODEL_NAME`; no-key environments should use `--mode fake`.

examples/optimization/eval_optimize_loop/agent/__init__.py

@@ -0,0 +1 @@
+"""Agent bridge package for the eval_optimize_loop example."""

examples/optimization/eval_optimize_loop/agent/agent.py

@@ -0,0 +1,142 @@
+# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
+#
+# Copyright (C) 2026 Tencent. All rights reserved.
+#
+# tRPC-Agent-Python is licensed under Apache-2.0.
+"""Live agent bridge for the eval_optimize_loop example.
+
+The optimizer contract is intentionally small: ``call_agent`` is an async
+function that accepts one user query and returns the final response text. This
+module re-reads the prompt file on every invocation so prompt candidates written
+by AgentOptimizer take effect immediately.
+
+The public bridge in this file mirrors the SDK docs:
+
+* ``create_agent`` builds a fresh ``LlmAgent`` from the current prompt file.
+* ``run_agent`` drives that agent through ``Runner`` and ``InMemorySessionService``.
+* ``make_call_agent`` returns the exact async callable required by
+  ``AgentOptimizer.optimize`` when a ``TargetPrompt`` is registered.
+"""
+
+from __future__ import annotations
+
+import os
+import uuid
+from pathlib import Path
+from typing import Any
+from typing import Awaitable
+from typing import Callable
+
+from trpc_agent_sdk.agents import LlmAgent
+from trpc_agent_sdk.models import OpenAIModel
+from trpc_agent_sdk.runners import Runner
+from trpc_agent_sdk.sessions import InMemorySessionService
+from trpc_agent_sdk.tools import FunctionTool
+from trpc_agent_sdk.types import Content
+from trpc_agent_sdk.types import Part
+
+
+APP_NAME = "eval_optimize_loop"
+
+
+def lookup_order(order_id: str) -> str:
+    """FunctionTool body used by the live ``LlmAgent`` example."""
+    data = {
+        "A100": "Order A100 is in transit and arrives on Friday.",
+        "A200": "Order A200 is delivered.",
+    }
+    return data.get(order_id, f"No order record found for {order_id}.")
+
+
+def search_policy(topic: str) -> str:
+    """FunctionTool body for policy and warranty lookup examples."""
+    topic_lower = topic.lower()
+    if "damaged" in topic_lower or "refund" in topic_lower:
+        return "Damaged items are eligible for a full refund within 30 days."
+    if "model z" in topic_lower or "warranty" in topic_lower:
+        return "Model Z has a 24-month warranty."
+    return "No matching policy snippet was found."
+
+
+def get_model_config() -> tuple[str, str, str]:
+    """Read live model credentials consumed by ``OpenAIModel``."""
+    api_key = os.getenv("TRPC_AGENT_API_KEY", "")
+    base_url = os.getenv("TRPC_AGENT_BASE_URL", "")
+    model_name = os.getenv("TRPC_AGENT_MODEL_NAME", "")
+    if not api_key or not base_url or not model_name:
+        raise ValueError(
+            "Live mode requires TRPC_AGENT_API_KEY, TRPC_AGENT_BASE_URL, and "
+            "TRPC_AGENT_MODEL_NAME. Use --mode fake for the no-key path."
+        )
+    return api_key, base_url, model_name
+
+
+def create_agent(prompt_path: Path) -> LlmAgent:
+    """Create a fresh ``LlmAgent`` from the current prompt file.
+
+    Re-reading here is the critical TargetPrompt contract: when
+    ``AgentOptimizer`` writes a candidate prompt, the next call immediately uses
+    that candidate without restarting the process.
+    """
+    api_key, base_url, model_name = get_model_config()
+    instruction = Path(prompt_path).read_text(encoding="utf-8").strip()
+    return LlmAgent(
+        name="support_assistant",
+        description="A support assistant whose system prompt is under optimization.",
+        model=OpenAIModel(model_name=model_name, api_key=api_key, base_url=base_url),
+        instruction=instruction,
+        tools=[FunctionTool(lookup_order), FunctionTool(search_policy)],
+    )
+
+
+async def run_agent(query: str, prompt_path: Path) -> dict[str, Any]:
+    """Run the live agent once and collect final text plus tool calls.
+
+    ``AgentOptimizer.optimize`` only needs final response text, but the outer
+    issue-level report also wants key trajectory information. This richer helper
+    supports both.
+    """
+    agent = create_agent(prompt_path)
+    session_service = InMemorySessionService()
+    runner = Runner(app_name=APP_NAME, agent=agent, session_service=session_service)
+    session_id = str(uuid.uuid4())
+    user_id = "optimizer"
+    await session_service.create_session(
+        app_name=APP_NAME,
+        user_id=user_id,
+        session_id=session_id,
+        state={},
+    )
+    message = Content(role="user", parts=[Part.from_text(text=query)])
+    final_text = ""
+    tools: list[dict[str, Any]] = []
+    async for event in runner.run_async(
+        user_id=user_id,
+        session_id=session_id,
+        new_message=message,
+    ):
+        if not event.content or not event.content.parts:
+            continue
+        for part in event.content.parts:
+            function_call = getattr(part, "function_call", None)
+            if function_call is not None:
+                tools.append(
+                    {
+                        "name": getattr(function_call, "name", None),
+                        "args": dict(getattr(function_call, "args", {}) or {}),
+                    }
+                )
+        if event.is_final_response():
+            for part in event.content.parts:
+                if getattr(part, "text", None) and not getattr(part, "thought", False):
+                    final_text += part.text
+    return {"text": final_text.strip(), "tools": tools}
+
+
+def make_call_agent(prompt_path: Path) -> Callable[[str], Awaitable[str]]:
+    """Return the fixed async ``(query: str) -> str`` bridge required by GEPA."""
+
+    async def call_agent(query: str) -> str:
+        return (await run_agent(query=query, prompt_path=prompt_path))["text"]
+
+    return call_agent

examples/optimization/eval_optimize_loop/case_meta.json

@@ -1,9 +1,35 @@
 {
-  "_comment": "Per-case metadata kept out of the SDK evalset so EvalSet stays schema-clean. 'key' marks cases that must not regress (gate uses it for critical-regression). 'rubric' selects the rubric dimension scored by run.score_case. 'tool_intent' is a trace-level attribution hint: 'authoritative_search' means the expected trajectory relies on the authoritative search tool, so a wrong tool is attributed to knowledge-recall insufficiency rather than a generic tool error.",
-  "train_ip_lookup_optimizable": {"key": false, "rubric": "none"},
-  "train_calendar_optimizable": {"key": false, "rubric": "none"},
-  "train_strict_json_ineffective": {"key": false, "rubric": "json_format"},
-  "val_search_fallback_new_pass": {"key": false, "rubric": "none", "tool_intent": "authoritative_search"},
-  "val_smalltalk_no_tool_regression": {"key": true, "rubric": "no_tool"},
-  "val_weather_soft_degradation": {"key": true, "rubric": "single_tool"}
+  "_comment": "Per-case metadata for attribution, gate checks, and fake/live trace scoring. It is kept outside evalsets so EvalSet schema validation remains clean.",
+  "train_order_lookup_optimizable": {
+    "category": "tool_call_error",
+    "key": false,
+    "rubric": "none"
+  },
+  "train_refund_policy_optimizable": {
+    "category": "knowledge_recall_insufficient",
+    "key": false,
+    "rubric": "none",
+    "authoritative_tool": "search_policy"
+  },
+  "train_json_format_ineffective": {
+    "category": "format_error",
+    "key": false,
+    "rubric": "json_format"
+  },
+  "val_warranty_new_pass": {
+    "category": "knowledge_recall_insufficient",
+    "key": false,
+    "rubric": "none",
+    "authoritative_tool": "search_policy"
+  },
+  "val_smalltalk_regression": {
+    "category": "spurious_tool_call",
+    "key": true,
+    "rubric": "no_tool"
+  },
+  "val_order_soft_degradation": {
+    "category": "spurious_tool_call",
+    "key": true,
+    "rubric": "single_tool"
+  }
 }