trpc-group · EricChan2028 · Jul 6, 2026
diff --git a/examples/optimization/eval_optimize_loop/DESIGN.md b/examples/optimization/eval_optimize_loop/DESIGN.md
@@ -0,0 +1,31 @@
+# 方案设计说明（Eval-Optimize Loop）
+
+**失败归因**：对每条失败 case 依据框架 metric 结果做规则归因，聚成六类——工具调用
+错误、工具参数错误、知识召回不足、格式不符合要求、LLM rubric 不达标、最终回复
+不匹配。轨迹失败先比调用名字多重集：名字不同判调用错误，名字同而参数异判参数
+错误，漏调知识工具时并报召回不足；期望回答可解析为 JSON 而实际不能则判格式违规。
+每条归因附证据与中文解释，规则未覆盖的失败按 metric 兜底映射，保证每个失败 case
+至少有一条可解释理由；根因按「轨迹在上游」的优先级选取。
+
+**接受策略**：候选须连过六道可配置闸门——验证集通过率与平均分双阈值提升、不得
+新增 hard fail、保护 case 不得退化、过拟合守卫、成本预算、时长预算，全部通过才
+接受；拒绝理由按严重度点名最关键的闸门，并列出同时未过的其它闸门。
+
+**防过拟合**：优化器只见弱指标（黑盒模式禁用轨迹与召回）与自己那份调参集；
+pipeline 一律用独立验证集加完整验收套件复评，出现「训练集提升且验证集退化」即判
+过拟合并拒绝，保护 case 与新增失败两道闸门再兜底。示例内置「泄漏调参集」场景：
+优化器视角一路变好，独立复评当场揭穿，守卫必将其拒绝。
+
+**产物审计**：每轮候选 prompt、接受理由、成本、耗时、种子与配置快照由优化器
+落盘 rounds/ 等目录；pipeline 另存基线与候选的逐 case 记录、归因明细、闸门配置
+快照，报告以 JSON 与 Markdown 双格式输出，整条链路离线确定可复现、可追溯审计。
+
+---
+
+**English abstract** — The loop evaluates baseline on train+val with the full metric
+suite, clusters failures into six explainable types, runs GEPA-based prompt
+optimization, then re-evaluates the candidate on an independent validation set with
+per-case deltas. Six configurable gates (improvement, no new hard fails, protected
+cases, overfit guard, cost and duration budgets) must all pass before acceptance,
+and every round leaves reproducible audit artifacts (candidates, reasons, cost,
+seed, config snapshots) in JSON and Markdown reports.
diff --git a/examples/optimization/eval_optimize_loop/README.en.md b/examples/optimization/eval_optimize_loop/README.en.md
@@ -0,0 +1,243 @@
+# Eval-Optimize Loop — Automated Evaluate · Attribute · Optimize · Regress · Audit
+
+[中文版](README.md) ｜ Design note: [DESIGN.md](DESIGN.md)
+
+> **Runs with zero API keys**: `python run_pipeline.py --scenario all` — all three
+> scenarios finish end-to-end in under a minute.
+
+## 1 · Problem & Design Goals
+
+`AgentOptimizer` can produce prompts with higher scores, but "higher score" does not
+mean "safe to ship":
+
+- the optimizer may only see **weak metrics** (black-box mode has no tool trajectory /
+  knowledge-recall signal);
+- if the optimizer's tuning set leaks from the training distribution it will
+  **overfit** without noticing;
+- without per-case comparison you cannot tell whether the gain was paid for by
+  breaking previously-passing key cases;
+- without audit artifacts, an improved prompt still cannot pass a production review.
+
+This example wires `AgentEvaluator` and `AgentOptimizer` into a **reproducible
+six-stage closed loop** that answers one question: **is this candidate prompt worth
+accepting?**
+
+```
+① baseline eval            ② failure attribution    ③ optimization
+train+val × 4 metrics  →   clustering into 6     →  AgentOptimizer (GEPA)
+(per-case score/traj)      failure types            (2 TargetPrompt fields)
+                                                          │
+⑥ audit artifacts      ⑤ acceptance gates        ④ candidate validation
+report json+md     ←   6 configurable gates  ←   independent re-eval
+per-round cand/cost    (all must pass)           per-case delta
+```
+
+### The minimal demo
+
+A "City Info Assistant": distance conversion (`convert_distance` tool), city
+introductions (`knowledge_search` tool + source citation), identity questions (no
+tool). The baseline prompt has three defects (unnormalized unit, no knowledge
+retrieval, non-JSON output); 6 committed cases (3 train + 3 val) expose all of them.
+Three built-in scenarios cover the three canonical outcomes:
+
+| Scenario | Candidate proposed by the optimizer | Independent re-eval | Gate decision |
+| --- | --- | --- | --- |
+| `success` | fixes all three defects | train 1/3→3/3, val 1/3→3/3 | ✅ ACCEPT |
+| `no_effect` | copy-editing only (directives unchanged) | all unchanged | ❌ REJECT (insufficient improvement) |
+| `overfit` | memorizes training samples | train 1/3→3/3, **val 1/3→0/3** | ❌ REJECT (overfit guard) |
+
+## 2 · Terminology
+
+| Term | Meaning |
+| --- | --- |
+| Acceptance suite | `data/eval_config.json`: trajectory + exact response + rubric + knowledge recall (4 metrics), used for regression eval |
+| Optimizer's weak metric | `optimizer.json` has only `final_response_avg_score`: the SDK forbids trajectory/recall metrics in black-box `call_agent` mode — this information gap is part of the overfitting story |
+| Leaked tuning set | `data/optimizer_probe.evalset.json`: same distribution as train; fed to the optimizer as its "validation set" only in the overfit scenario |
+| Protected cases | `protected_cases` in `pipeline.json`: whitelist of key cases; any regression rejects the candidate |
+| Directive DSL | the `<!-- directives: ... -->` comment block in prompts; the fake agent parses it, so prompt edits change behavior offline for real |
+| Trace mode | `evalMode: "trace"` in the evalset: evaluate/attribute pre-recorded traces without running the agent |
+
+## 3 · Running
+
+### 3.1 Zero-dependency run (no env vars / API keys)
+
+```bash
+# default scenario: success
+python examples/optimization/eval_optimize_loop/run_pipeline.py
+
+# all three scenarios (recommended first run)
+python examples/optimization/eval_optimize_loop/run_pipeline.py --scenario all
+
+# additionally evaluate/attribute the pre-recorded baseline traces (trace mode)
+python examples/optimization/eval_optimize_loop/run_pipeline.py --baseline-from-trace
+
+# write the best candidate back to loop_agent/prompts/ when gates pass (mutates sources!
+# with --scenario all the write is deferred until every scenario has finished)
+python examples/optimization/eval_optimize_loop/run_pipeline.py --apply
+
+# validate a report against the schema contract
+python examples/optimization/eval_optimize_loop/run_pipeline.py --check sample_output/success/optimization_report.json
+```
+
+Tests:
+
+```bash
+python -m pytest examples/optimization/eval_optimize_loop/tests -q
+```
+
+### 3.2 Output layout
+
+```
+runs/<scenario>-<timestamp>/
+├── optimization_report.json       # structured report: baseline / candidate / delta / attribution / gate decision
+├── optimization_report.md         # human-readable verdict with all the evidence
+├── baseline_eval.json             # stage ① raw per-case records
+├── candidate_eval.json            # stage ④ raw per-case records
+├── attribution.json               # stage ② findings
+├── pipeline_config.snapshot.json  # gate/seed config snapshot of this run
+└── optimize/                      # stage ③ native SDK audit directory
+    ├── result.json  summary.txt  run.log  config.snapshot.json
+    ├── rounds/round_001.json …    # per-round candidate prompts, acceptance reason, cost, duration
+    └── baseline_prompts/  best_prompts/
+```
+
+The committed `sample_output/` holds the three reports from `--scenario all`; to
+regenerate:
+
+```bash
+python examples/optimization/eval_optimize_loop/run_pipeline.py --scenario all --output /tmp/regen
+# then copy /tmp/regen/<scenario>-*/optimization_report.{json,md} into sample_output/<scenario>/
+```
+
+## 4 · Inputs / Outputs
+
+| File | Role |
+| --- | --- |
+| `data/train.evalset.json` | 3 training cases (reflection minibatch source) |
+| `data/val.evalset.json` | 3 independent validation cases (final referee for regression) |
+| `data/optimizer_probe.evalset.json` | 3 leaked tuning cases (fed to the optimizer only in `overfit`) |
+| `data/trace_baseline.evalset.json` | 2 trace-mode cases (pre-recorded baseline failures) |
+| `data/eval_config.json` | acceptance metric suite (4 metrics, fake judge) |
+| `optimizer.json` / `configs/optimizer.*.json` | optimizer configs (scenarios differ only in `reflection_lm.model_name`) |
+| `pipeline.json` | gate thresholds / protected cases / budgets / seed |
+| `loop_agent/prompts/system.md`, `skill.md` | the two TargetPrompt source files |
+| `candidates/*.md` | the fake reflection LM's proposal library (scenario × field) |
+| `sample_output/*/optimization_report.{json,md}` | sample reports for the three scenarios |
+
+## 5 · Gate rules (stage ⑤)
+
+All six gates must pass; the rejection reason cites the most severe failed gate
+(overfit > protected case > new hard fail > insufficient improvement > budgets).
+Every threshold is configurable in `pipeline.json`.
+
+| Gate | Rule | Config |
+| --- | --- | --- |
+| `min_val_improvement` | val pass-rate gain ≥ threshold AND mean-score gain ≥ threshold | `min_val_pass_rate_improvement` / `min_val_score_improvement` |
+| `no_new_hard_fail` | no case may flip pass→fail | `forbid_new_hard_fail` |
+| `protected_cases` | any new_fail / score_down on a protected case rejects | `protected_cases` |
+| `overfit_guard` | train pass-rate ↑ AND val pass-rate ↓ ⇒ overfitting | `overfit_guard` |
+| `cost_budget` | optimization cost ≤ budget; metric calls ≤ budget (optional) | `max_cost_usd` / `max_metric_calls` |
+| `duration_budget` | pipeline wall-clock ≤ budget | `max_duration_seconds` |
+
+The 12-row decision matrix lives in `tests/test_gates.py::DECISION_MATRIX` (12/12 pass).
+
+## 6 · Failure-attribution rules (stage ②)
+
+The rules depend only on the structure of framework metric results, not on this
+example's cases (they generalize to hidden samples); every failing case is
+**guaranteed at least one** evidence-backed reason (a per-metric fallback mapping
+covers anything the rules miss).
+
+| Failure type | Trigger |
+| --- | --- |
+| `wrong_tool_call` | trajectory metric failed and the actual/expected call **name multisets** differ (missing/extra/wrong tool) |
+| `wrong_tool_args` | trajectory metric failed with equal names but different arguments |
+| `knowledge_recall_miss` | recall rubric failed; or the missing call is a knowledge tool (reported alongside the trajectory finding) |
+| `format_violation` | response metric failed and the expected text parses as JSON while the actual does not |
+| `llm_rubric_fail` | response-quality rubric failed (evidence = failing rubric ids + reasons) |
+| `final_answer_mismatch` | any other response-metric failure |
+
+Primary (root-cause) precedence: `wrong_tool_call` > `wrong_tool_args` >
+`knowledge_recall_miss` > `format_violation` > `llm_rubric_fail` >
+`final_answer_mismatch` (trajectory errors sit upstream in the chain).
+
+## 7 · Design notes
+
+### 7.1 Optimizer metrics ≠ acceptance metrics (a deliberate information gap)
+
+Black-box `call_agent` mode cannot capture tool trajectories or tool responses, and
+the SDK **hard-rejects** `tool_trajectory_avg_score` / `llm_rubric_knowledge_recall`
+in that mode. So `optimizer.json` carries only exact response matching — exactly the
+situation in real businesses where the optimizer sees a weaker signal than the
+acceptance suite. That is why the loop exists: **the optimizer's claim of improvement
+only counts after re-evaluation with the full suite on an independent validation
+set.** The overfit scenario pushes this gap to the extreme (optimizer view 0/3→3/3,
+independent re-eval val 1/3→0/3).
+
+### 7.2 How the three fake models run with zero API keys (no SDK changes)
+
+Judge and reflection-LM configs both support `provider_name`; any non-openai provider
+routes through `ModelRegistry.create_model("{provider}/{model}")` regex matching. The
+example registers three deterministic fake providers:
+
+- **fake agent** parses the directive DSL in the prompt and changes behavior
+  accordingly — prompt edits produce real behavior/score differences offline;
+- **fake judge** evaluates rubric text via backtick tokens; its conditional rule
+  ("如果…" + condition-not-applicable ⇒ yes) mirrors the real judge prompt, and its
+  JSON output is isomorphic to the real judge's, parsed by the SDK's own scorer;
+- **fake reflection** finds the `<!-- prompt-field: X -->` marker in the reflection
+  request and returns `candidates/X.<scenario>.md`, the scenario taken from its own
+  `model_name`.
+
+### 7.3 Why the packages are named `loop_agent` / `loop_pipeline`
+
+pytest may import several examples in one process: `agent` is used by most examples
+and `pipeline` is already taken by `multi_agent_pipeline`; duplicate top-level names
+clobber each other in `sys.modules`.
+
+### 7.4 Semantics of `--apply`
+
+The optimizer itself always runs with `update_source=False` (sources are restored
+when it finishes). Writing the best candidate back to `loop_agent/prompts/` requires
+**both** an accepting gate decision **and** the explicit `--apply` flag, so a
+rejected candidate can never reach the source files, and even accepted ones land in
+the audit directory by default.
+
+## 8 · Adapting to your own business
+
+1. **Swap the agent**: replace `loop_agent/`, keeping the two entry points —
+   `get_agent_async()` (evaluator agent_module mode, captures tool trajectories) and
+   `call_agent(query) -> str` (optimizer black-box callback). Both must re-read the
+   prompt sources on every call.
+2. **Swap the data**: point `data/train.evalset.json` / `data/val.evalset.json` at
+   your business cases. **The validation set must be independent of training** (the
+   SDK guards against same-file leakage; same-distribution-different-file leakage is
+   what this loop's overfit guard is for).
+3. **Swap the acceptance suite**: configure a real `judge_model` in
+   `data/eval_config.json` (drop `provider_name: fake-judge`, set
+   `model_name`/`api_key`/`base_url`).
+4. **Swap the optimizer config**: same for `reflection_lm` in `optimizer.json`;
+   black-box mode allows response-based metrics only.
+5. **Tune the gates**: adjust `pipeline.json` to your risk profile — add key
+   regression cases to `protected_cases`, set `max_cost_usd` from real model pricing.
+
+## 9 · FAQ
+
+**Q: Why does the `no_effect` scenario report `SUCCEEDED` yet get rejected?**
+`OptimizeResult.status=SUCCEEDED` only means the loop terminated normally
+(`finish_reason=no_improvement`). Acceptance is the pipeline gates' job — the two
+verdict layers are deliberately separate.
+
+**Q: Why does the overfit guard require "train↑ AND val↓" instead of just val↓?**
+A val drop alone only says the candidate is bad; a simultaneous train rise is the
+fingerprint of overfitting, letting the report give an actionable diagnosis ("your
+tuning set leaks from training") instead of a generic "it got worse".
+
+**Q: When is trace mode useful?**
+When you already have production trace logs and want attribution before deciding to
+run optimization: `--baseline-from-trace` evaluates and attributes
+`data/trace_baseline.evalset.json` (`evalMode: "trace"`) with zero agent execution.
+
+**Q: Why is the reported cost 0?**
+Fake models incur no token cost. With real models, `OptimizeResult.total_llm_cost` /
+`total_token_usage` flow into the report and the `cost_budget` gate automatically.