feat(examples): add eval optimization loop

Author: xyxhhhhh

examples/optimization/eval_optimize_loop/.gitignore

@@ -0,0 +1,4 @@
+runs/
+__pycache__/
+*.pyc
+

examples/optimization/eval_optimize_loop/README.md

@@ -29,6 +29,30 @@ To refresh the committed sample reports:
 PYTHONPATH=../../.. python run_pipeline.py --mode fake --update-sample-outputs
 ```
 
+The fake and trace modes support three deterministic scenarios:
+
+```bash
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --scenario overfit
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --scenario accepted
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --scenario cost_exceeded
+```
+
+- `overfit` is the default: train improves, validation does not improve, and a
+  critical case regresses, so the gate rejects the candidate.
+- `accepted`: train and validation both improve with no new hard fail, so the
+  gate accepts the candidate.
+- `cost_exceeded`: quality improves, but the estimated cost exceeds
+  `gate.max_cost_usd`, so the gate rejects the candidate.
+
+For CI-style scripts, add `--ci-exit-code`. Normal demo runs always exit 0 after
+writing the report; CI mode exits 0 for accepted candidates and 1 for rejected
+candidates:
+
+```bash
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --scenario accepted --ci-exit-code
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --scenario overfit --ci-exit-code
+```
+
 Trace replay mode also runs without model credentials. It first records fake
 outputs into generated `eval_mode: "trace"` evalsets, then evaluates those
 `actual_conversation` records without invoking `call_agent`:
@@ -48,81 +72,166 @@ export TRPC_AGENT_MODEL_NAME="<your-model>"
 PYTHONPATH=../../.. python run_pipeline.py --mode optimizer
 ```
 
+## Verification Commands
+
+Use these commands to exercise the important review paths:
+
+```bash
+cd examples/optimization/eval_optimize_loop
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --scenario overfit
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --scenario accepted
+PYTHONPATH=../../.. python run_pipeline.py --mode fake --scenario cost_exceeded
+PYTHONPATH=../../.. python run_pipeline.py --mode trace --scenario overfit
+cd ../../..
+pytest tests/evaluation/test_eval_optimize_loop_example.py -q
+```
+
 ## Inputs
 
 | File | Role |
 | --- | --- |
 | `train.evalset.json` | Training split used for baseline scoring and optimization feedback. |
 | `val.evalset.json` | Validation split used for regression and gate decisions. |
 | `optimizer.json` | Shared metric, optimizer, fake candidate, and gate configuration. |
-| `agent/prompts/system.md` | Baseline prompt source registered as `TargetPrompt("system_prompt")`. |
+| `case_meta.json` | Public-case attribution labels used for local self-checking. |
+| `agent/prompts/router.md` | Router prompt source registered as `TargetPrompt("router_prompt")`. |
+| `agent/prompts/system.md` | System prompt source registered as `TargetPrompt("system_prompt")`. |
+| `agent/prompts/skill.md` | Skill prompt source registered as `TargetPrompt("skill_prompt")`. |
 
 The fake mode still uses `AgentEvaluator` for all train and validation scoring.
 Only candidate generation is deterministic, so the example can run in CI or on a
 laptop without model credentials.
 
+At startup, the pipeline validates that all required input files exist, train and
+validation evalsets are distinct and non-empty, gate keys are present, critical
+case IDs refer to validation cases, and evaluator parallelism is positive.
+
+Fake judge behavior is implemented by local exact-match final-response scoring
+(`final_response_avg_score`). The report records this as
+`run.judge_mode = "local_exact_match_fake_judge"`, so the evaluation path does
+not need an LLM judge or API key.
+
+For tool-only and rubric-only failure classes, fake judge mode uses
+deterministic attribution hints from `case_meta.json`. This lets the no-key
+example exercise the full attribution taxonomy without requiring real tool
+execution or an LLM rubric judge.
+
 ## Outputs
 
-Each run writes to `runs/<mode_timestamp>/`:
+Each run writes to `runs/<mode>_<scenario>_<timestamp>/` by default:
 
 | File or directory | Contents |
 | --- | --- |
-| `optimization_report.json` | Machine-readable baseline, candidate, delta, attribution, gate, and audit payload. |
+| `optimization_report.json` | Machine-readable baseline, candidate, per-round prompts, delta, attribution, gate, and audit payload. |
 | `optimization_report.md` | Human-readable report for review. |
 | `candidate_prompts/` | Candidate prompt text per target field. |
 | `eval_*` | Raw `AgentEvaluator` result files for every phase. |
 | `config.snapshot.json` | Reproducible copy of the input config. |
 | `eval_metrics.snapshot.json` | Evaluator-compatible metric config extracted from `optimizer.json.evaluate`. |
 
+The report also includes `prompt_audit`, which records each target prompt's
+source path, baseline SHA-256, candidate SHA-256, character counts, and whether
+the candidate changed the prompt. Each prompt audit entry also includes a capped
+unified diff preview, so reviewers can see what changed without opening the
+candidate prompt files. This makes the score-to-prompt relationship auditable
+even when reports are copied outside the run directory.
+
+`input_audit` records stable SHA-256 hashes and byte counts for the train
+evalset, validation evalset, optimizer config, case metadata, and prompt source
+files. This makes a report reproducible even after it is detached from the
+working tree.
+
+`audit.cost` splits the total estimated spend into optimizer and evaluation
+components, then applies the same total to the configurable budget gate. In
+fake/trace mode both components are deterministic and normally zero except for
+the dedicated `cost_exceeded` scenario.
+
+`failure_attribution.self_check` compares attribution results against the public
+gold labels in `case_meta.json`. Gold labels live in `expected_failure_types`.
+For categories that cannot naturally arise in the no-key black-box path, such as
+tool-call and LLM-rubric failures, `fake_attribution_hints` provides an explicit
+fake-judge hint. The report marks those cases as `hint_assisted` and also reports
+rule-only accuracy separately. This is not a hidden-set proof, but it provides a
+local accuracy guard for the bundled examples and fails report validation if the
+labeled-case accuracy drops below 75%.
+
 Committed examples live in `sample_outputs/`.
 
+The JSON report includes both human reasons and structured gate checks:
+
+```json
+{
+  "gate": {
+    "decision": "rejected",
+    "reasons": ["..."],
+    "checks": [
+      {"name": "min_validation_score_delta", "passed": false},
+      {"name": "no_new_hard_fail", "passed": false}
+    ]
+  }
+}
+```
+
+## Gate Decision Contract
+
+A candidate is accepted only when every configured gate check passes:
+
+| Check | Accept condition | Reject condition |
+| --- | --- | --- |
+| `min_validation_score_delta` | Validation score delta is greater than or equal to `gate.min_validation_score_delta`. | Validation score delta is below the configured threshold. |
+| `no_new_hard_fail` | No validation case changes from pass to fail, unless `gate.allow_new_hard_fail` is true. | At least one validation case becomes a new hard fail while new hard fails are disallowed. |
+| `critical_case_regression` | No case in `gate.critical_case_ids` loses score. | Any critical validation case regresses. |
+| `overfitting_guard` | Validation improves, or training does not improve. | Training improves while validation does not improve. |
+| `max_cost_usd` | Estimated candidate cost is less than or equal to `gate.max_cost_usd`. | Estimated candidate cost exceeds the budget. |
+
+This contract is intentionally validation-first: training improvements alone
+never justify acceptance, and hidden samples are expected to follow the same
+structured decision rules.
+
 ## Case Coverage
 
-The six sample cases cover the required outcomes:
+The ten sample cases exceed the minimum requirement and cover the required outcomes:
 
 | Case | Split | Expected behavior |
 | --- | --- | --- |
 | `train_refund_double_charge` | train | Candidate fixes a baseline failure. |
 | `train_password_reset` | train | Already passing; candidate is unchanged. |
 | `train_legacy_sync` | train | Candidate does not help. |
+| `train_plan_question` | train | Stable non-target billing question remains passing. |
+| `train_policy_tool_missing` | train | Tool lookup failure attribution with an incomplete JSON response. |
 | `val_vip_refund` | validation | Candidate creates a new pass. |
 | `val_plan_question` | validation | Candidate has no effect. |
 | `val_checkout_outage` | validation | Candidate regresses a critical case. |
+| `val_mobile_crash` | validation | Stable technical troubleshooting case remains passing. |
+| `val_rubric_tone_fail` | validation | Deterministic rubric failure attribution remains failing. |
 
 The default fake candidate improves train score but does not improve validation
 overall because it also introduces a new hard fail. The gate therefore rejects
 the candidate and records the overfitting reason.
 
+## Requirement Mapping
+
+| #91 requirement | Implementation |
+| --- | --- |
+| Baseline train/validation evaluation | `run_pipeline.py::evaluate_dataset` uses `AgentEvaluator.get_executer(...).get_result()`. |
+| Per-case metric, pass/fail, reason, trace | Report cases include query, expected, actual, metric details, status, failure types, and reason. |
+| Failure attribution clustering | `attribute_failure` and `build_failure_stats`. |
+| Full attribution taxonomy | Public sample failures include `final_response_mismatch`, `tool_call_error`, `parameter_error`, `llm_rubric_not_met`, `knowledge_recall_insufficient`, and `format_violation`. |
+| Attribution accuracy guard | `case_meta.json` plus `failure_attribution.self_check` validates public labeled cases and marks fake hint-assisted cases. |
+| Prompt optimization target | `TargetPrompt` registers `router_prompt`, `system_prompt`, and `skill_prompt`. |
+| AgentOptimizer or equivalent extension | `--mode optimizer` delegates to `AgentOptimizer`; fake/trace modes use deterministic candidate generation for no-key CI. |
+| Candidate validation and per-case delta | `build_case_deltas` marks `new_pass`, `new_fail`, `score_improved`, `score_regressed`, and `unchanged`. |
+| Configurable gate | `optimizer.json.gate` plus `apply_gate`, including structured `gate.checks`. |
+| Overfitting rejection | Default `--scenario overfit` improves train but rejects because validation does not improve and a critical case regresses. |
+| Cost budget rejection | `--scenario cost_exceeded` improves quality but fails `max_cost_usd`. |
+| Audit artifacts | `input_audit`, `prompt_audit` with diff previews, `optimization_rounds`, `candidate_prompts/`, raw `eval_*` outputs, config snapshots, JSON report, and Markdown report. |
+| No API key mode | `--mode fake` and `--mode trace`. |
+| 6 public cases | `train.evalset.json` has 5 cases and `val.evalset.json` has 5 cases. |
+
 ## Design Notes
 
-The pipeline treats evaluation results as the source of truth and converts
-`AgentEvaluator.get_executer(...).get_result()` into a stable per-case report.
-For each split, it records score, pass/fail status, normalized actual and
-expected final responses, metric details, and a short trace summary. Failure
-attribution is deterministic: invalid JSON or missing keys are classified as
-format violations; category mismatches indicate knowledge-recall gaps; priority
-mismatches are parameter errors; action mismatches are final-response errors.
-This keeps fake mode explainable and also gives real optimizer runs a fallback
-when no LLM judge reason is available.
-
-Candidate prompts are validated on the held-out validation set before any source
-prompt is updated. The configurable gate checks minimum validation improvement,
-new hard failures, critical-case regressions, and cost budget. It also includes
-an explicit overfitting guard: when training improves but validation does not,
-the candidate is rejected even if some individual validation case improves. All
-decisions include concrete reasons, so a reviewer can see whether rejection came
-from score, hard-fail, critical-case, cost, or overfitting policy.
-
-Auditability is handled by snapshotting config, writing candidate prompt files,
-preserving raw evaluator outputs, and emitting both JSON and Markdown reports.
-The JSON report is suitable for CI parsing, while the Markdown report summarizes
-score deltas, per-case outcomes, failure clusters, and the final accept/reject
-decision for humans. Fake mode makes this loop reproducible without API keys;
-real mode can swap in `AgentOptimizer` while keeping the same validation, gate,
-and report machinery. The source prompt is never overwritten by this example;
-reviewers can inspect `candidate_prompts/` and decide whether to copy or submit
-the change through a normal PR. Hidden samples can reuse the same gate because
-all decisions are based on held-out validation deltas rather than train-set
-scores alone. Trace-style replay can also be added by replacing `call_agent`
-with eval cases that provide `actual_conversation`, while the attribution, delta,
-gate, and report stages remain unchanged.
+本示例把 `AgentEvaluator.get_executer(...).get_result()` 作为唯一评测事实来源，分别对训练集和验证集生成逐 case 记录，包含输入、期望输出、实际输出、metric 分数、pass/fail、失败类型和原因。失败归因采用可复现规则：JSON 解析失败或缺字段归为格式不符合要求，category 不一致归为知识召回不足，priority 不一致归为参数错误，action 不一致归为最终回复不匹配；对必须依赖真实工具轨迹或 LLM rubric judge 的类别，fake mode 通过 `case_meta.json` 的 `fake_attribution_hints` 标记 `tool_call_error` 和 `llm_rubric_not_met`，并在 self-check 中区分 hint-assisted 与 rule-only 准确率，避免把公开标签直接当成黑盒判断结果。
+
+候选 prompt 不会直接写回源文件，而是先在验证集回归，并和 baseline 做逐 case delta，标记新增通过、新增失败、分数提升、分数下降和无变化。gate 由 `optimizer.json` 配置，检查验证集总分提升阈值、是否新增 hard fail、关键 case 是否退化以及成本预算；额外防过拟合规则是训练集提升但验证集没有提升时拒绝候选。这样即使优化器记住训练失败样例，只要验证集没有真实收益，或者牺牲了关键 case，也不会进入可接受状态。
+
+审计产物包括配置快照、随机种子、候选 prompt 文件、`optimization_rounds`、原始 evaluator 输出、JSON 报告和 Markdown 报告，便于完整复现实验配置，也便于后续 CI 读取结构化字段。`prompt_audit` 记录源路径、baseline/candidate 哈希、字符数和是否变更，reviewer 可以把分数变化追溯到具体 prompt。trace mode 会生成 `eval_mode: "trace"` 数据并回放 `actual_conversation`，保证无 API key 环境也能复现 baseline、归因、优化、验证、gate 和审计闭环。

examples/optimization/eval_optimize_loop/agent/agent.py

@@ -18,7 +18,10 @@
 from pathlib import Path
 
 
-SYSTEM_PROMPT_PATH = Path(__file__).parent / "prompts" / "system.md"
+PROMPT_DIR = Path(__file__).parent / "prompts"
+ROUTER_PROMPT_PATH = PROMPT_DIR / "router.md"
+SYSTEM_PROMPT_PATH = PROMPT_DIR / "system.md"
+SKILL_PROMPT_PATH = PROMPT_DIR / "skill.md"
 
 _SPACE_RE = re.compile(r"\s+")
 
@@ -36,8 +39,15 @@ def normalize_json_text(raw: str) -> str:
     return _compact_json(parsed)
 
 
+def _read_prompt_text() -> str:
+    return "\n\n".join(
+        path.read_text(encoding="utf-8")
+        for path in (ROUTER_PROMPT_PATH, SYSTEM_PROMPT_PATH, SKILL_PROMPT_PATH)
+    ).lower()
+
+
 def _prompt_flags() -> dict[str, bool]:
-    prompt = SYSTEM_PROMPT_PATH.read_text(encoding="utf-8").lower()
+    prompt = _read_prompt_text()
     return {
         "refund_rule": "treat double charge" in prompt or "vip refund requests" in prompt,
         "keep_outage_p1": "p1 for urgent production outages" in prompt,
@@ -106,6 +116,19 @@ def answer_for_query(query: str) -> str:
             "action": "answer",
         })
 
+    if "policy citation" in text or "pol-77" in text:
+        return _compact_json({
+            "category": "billing",
+            "action": "answer",
+        })
+
+    if "guaranteed instant fix" in text or "repeated invoice confusion" in text:
+        return _compact_json({
+            "category": "billing",
+            "priority": "p3",
+            "action": "answer",
+        })
+
     if "mobile app crashes" in text:
         return _compact_json({
             "category": "technical",

examples/optimization/eval_optimize_loop/agent/prompts/router.md

@@ -0,0 +1,7 @@
+Route each support ticket to one primary handling path before answering.
+
+Baseline routing rules:
+- Account access and profile questions go to account handling.
+- Billing and pricing questions go to billing handling.
+- Production incidents and software failures go to technical handling.
+

examples/optimization/eval_optimize_loop/agent/prompts/skill.md

@@ -0,0 +1,9 @@
+Apply these triage skills after routing:
+
+- Use p1 only when the ticket describes an urgent production outage.
+- Use p2 for normal user-impacting issues.
+- Use p3 for low-risk informational requests.
+- Use answer for informational account or pricing questions.
+- Use escalate for production incidents.
+- Use troubleshooting for reproducible technical failures.
+

examples/optimization/eval_optimize_loop/case_meta.json

@@ -0,0 +1,24 @@
+{
+  "train_refund_double_charge": {
+    "expected_failure_types": ["final_response_mismatch", "knowledge_recall_insufficient"],
+    "note": "Baseline routes refund billing work to account/answer instead of billing/refund_review."
+  },
+  "train_legacy_sync": {
+    "expected_failure_types": ["final_response_mismatch"],
+    "note": "Baseline returns troubleshooting while the reference expects escalation."
+  },
+  "train_policy_tool_missing": {
+    "expected_failure_types": ["tool_call_error", "format_violation"],
+    "fake_attribution_hints": ["tool_call_error"],
+    "note": "The no-key fake agent cannot perform the expected policy lookup and returns an incomplete JSON object."
+  },
+  "val_vip_refund": {
+    "expected_failure_types": ["final_response_mismatch", "knowledge_recall_insufficient", "parameter_error"],
+    "note": "Baseline misses billing/refund_review and p1 priority for VIP refund."
+  },
+  "val_rubric_tone_fail": {
+    "expected_failure_types": ["llm_rubric_not_met", "parameter_error"],
+    "fake_attribution_hints": ["llm_rubric_not_met"],
+    "note": "The answer uses a lower urgency than the reference and is deterministically labeled as a rubric failure for the no-key fake judge path."
+  }
+}