docs(examples): finalize issue 91 closed loop

Author: YAO-001

examples/optimization/eval_optimize_loop/DESIGN.md

@@ -1,16 +1,9 @@
 # 设计说明
 
-## 失败归因
-评测先由 FakeJudge 产出结构化失败，再用规则归因：JSON、禁用格式归为 `format_violation`，精确答案错归为 `final_response_mismatch`，工具名、参数、知识召回、长度和 rubric 各有独立类别。每个失败都保留 reason 与 evidence，便于复核。若样例声明期望类别，报告会计算归因准确率。
+本示例把基线评测、失败归因、提示词优化、候选复评、接受门禁和审计落盘串成同一条可复现流水线。训练集与验证集采用 SDK 标准 EvalSet；FakeModel 只读取提示词和用户输入，不能接触期望答案、标签、受保护标记或样例编号，真值仅供评测器、归因器和门禁使用，从源头避免答案泄漏。
 
-## 接受门禁
-Gate 要求验证集提升达到阈值，并检查新硬失败、受保护样例退化、单例降分和累计成本。若训练分上涨但验证不涨，直接标记过拟合并拒绝。
+失败结果按格式、最终回复、工具调用、参数、知识召回和规则评分等类别聚合，每个失败样例保留原因与证据。训练集只用于暴露问题，所有去重后的候选都必须重新执行完整训练集和验证集评测，再逐样例计算新增通过、新增失败、升分和降分。统一门禁检查验证集增益、过拟合、新增硬失败、受保护样例退化、单例降分和累计成本；总成本不完整时，任何数值预算都以 `cost_unavailable` 拒绝。
 
-## 防过拟合
-训练集只用于暴露问题，候选必须通过验证集和 protected case。过拟合候选即使修好训练格式，只要把验证集自然语言或受保护精确答案改坏，也会被拒绝。`delta_type` 标出 new_pass、new_fail、score_up、score_down，避免只看平均分掩盖局部退化。
+SDK 优化始终使用 `update_source=False`，并记录真实逐样例指标、失败证据及轨迹是否可用。只有候选通过完整门禁、输入完整性检查且预写审计成功后，`--update-source` 才由共享流水线事务性回写所选提示词；失败时保留原始字节或明确报告恢复错误。
 
-## Fake 与 SDK
-Fake mode 由 expectation、tags、protected 和 simulated_outputs 驱动，不依赖样例 id，可无 API key 稳定复现。SDK mode 通过 `SDKBackend` 调用 `AgentOptimizer` 与 `TargetPrompt`，失败时给出明确错误，不回退到 fake。
-
-## 审计与回写
-报告同时写 JSON、Markdown 和 `runs/<run_id>/`，保存输入哈希、配置快照、候选 prompt、diff、case 结果与成本，并给出可复现命令。默认不回写源 prompt，只有显式 `--update-source` 才允许，报告会记录该选择。
+运行产物先写入 `runs/.<run_id>.tmp/`，包含基线与候选提示词、每轮记录、全部分割评测结果、逐样例差异、门禁、成本和回写日志。发布时采用不可覆盖的原子目录重命名：Windows 使用 MoveFileExW，Linux 使用 renameat2；其他缺少安全原语的平台直接失败。最终目录不可变，清单记录每个文件的 SHA-256 与大小，顶层报告仅是发布后的便利副本。

examples/optimization/eval_optimize_loop/README.md

@@ -17,7 +17,7 @@ loader.py / config.py  --> validated cases, gate config, input hashes
         v
 backends.py
   |-- FakeBackend -> FakeModel + FakeJudge + FakeOptimizer
-  `-- SDKBackend  -> AgentOptimizer + TargetPrompt + user call_agent
+  `-- SDKBackend  -> AgentOptimizer + TargetPrompt + user call_agent + AgentEvaluator
         |
         v
 attribution.py -> evaluator.py -> gate.py -> report.py
@@ -87,47 +87,59 @@ python examples/optimization/eval_optimize_loop/run_pipeline.py \
 needed by that callable. SDK mode never silently falls back to fake mode. The
 generated reproducibility command records the actual `--sdk-call-agent`
 `module:function` target and file/config paths, but it does not record API keys
-or other provider secrets.
+or other provider secrets. Persisted optimizer config snapshots recursively
+replace credential values with `<redacted>`.
+
+The checked-in `train.evalset.json` and `val.evalset.json` use the SDK
+`EvalSet` schema (`eval_set_id` / `eval_cases`). Fake mode reads its
+deterministic judge metadata from each case's `session_input.state`
+(`eval_optimize_expectation`, `eval_optimize_tags`, and
+`eval_optimize_protected`).
 
 SDK optimizer config and wrapper gate config are intentionally separate.
 `--optimizer-config` is passed unchanged to `AgentOptimizer.optimize`, so it must
 follow the SDK `OptimizeConfigFile` schema. Put wrapper-only gate settings in
 `--gate-config` (for example `{"gate": {"min_val_score_improvement": 0.05,
-"max_total_cost": 1.0}}`). If `--gate-config` is omitted, the wrapper uses the
-same default aggregate gate values as the fake example.
+"allow_new_hard_fail": false, "max_score_drop_per_case": 0.0,
+"max_total_cost": null}}`). If `--gate-config` is omitted, the wrapper uses the
+same default gate values as fake mode and auto-adds validation cases marked with
+`eval_optimize_protected`. A numeric `max_total_cost` fails closed with
+`cost_unavailable` whenever the backend cannot prove a complete total. Use
+`null` only when you intentionally want to run the quality gates without a cost
+limit; `reported_optimizer_cost` remains an incomplete lower bound, not a total.
 
 `--target-prompt name=path` may be repeated. If omitted, SDK mode keeps the old
 single-field behavior and registers `system_prompt=--prompt`. A run can optimize
 only `router_prompt`, only `skill_prompt`, or any set of named fields as long as
 `OptimizeResult.best_prompts` returns every registered field.
 
-Fake mode is the complete per-case closed loop. SDK mode is the real
-`AgentOptimizer` / `TargetPrompt` path with an aggregate wrapper gate. When the
-SDK optimizer returns a best prompt, this wrapper maps `OptimizeResult`
-aggregate fields into the JSON/Markdown report: baseline/best pass rate,
-pass-rate improvement, metric breakdowns, token usage, duration, LLM cost,
-all `best_prompts`, and round summaries. SDK mode applies
-`gate_status: partial_applied`: it checks `status == SUCCEEDED`, validation
-improvement against `gate.min_val_score_improvement`, and total LLM cost against
-`gate.max_total_cost`. Protected-case regression, new-hard-failure, per-case
-delta, and per-case score-drop checks are not claimed in SDK mode unless the SDK
-exposes full per-case validation scores; they are listed in
-`not_applied_checks`.
-
-Fake mode uses a deterministic run id (`eval_optimize_loop_seed_<seed>`) so the
-example outputs are byte-stable. SDK mode is append-only by default: the wrapper
-derives a compact UTC `run.run_id` from the SDK result `started_at` when
-available, otherwise from the current UTC timestamp. Pass `--run-id` only when
-a fixed audit path is useful for tests or local smoke runs; only explicit
-`--run-id` values are included in the reproducibility command.
+Fake mode is the deterministic offline closed loop. SDK mode is the real
+`AgentOptimizer` / `TargetPrompt` path followed by explicit post-optimization
+`AgentEvaluator` runs. The adapter deduplicates every prompt bundle found in
+`OptimizeResult.rounds[].candidate_prompts` and `best_prompts`. The shared
+pipeline temporarily installs each bundle, reruns the complete train and
+validation evalsets, restores the exact source bytes, then computes the same
+per-case deltas and complete gate decision used by fake mode. SDK case results
+carry real per-metric scores, failure reason/evidence, and `trace_available`;
+missing SDK trace data is reported as unavailable rather than fabricated.
+Optimizer aggregates remain audit context and are never substituted for these
+post-optimization case results.
+
+Both modes default to a collision-resistant run id of the form
+`eval_optimize_loop_<mode>_<UTC timestamp>_<random suffix>`. Pass `--run-id`
+when a stable path is useful. Existing temporary or final run ids are never
+reused or overwritten. The checked-in example is generated in a fresh output
+directory with the explicit id `example`.
 
 ## Source Prompt Writes
 
-The default is **no source write-back**. The baseline prompt file is not modified
-by fake mode, and `SDKBackend` calls `AgentOptimizer.optimize(update_source=False)`
-unless `--update-source` is explicitly passed. The report records
-`run.update_source` and the Markdown report states whether source write-back was
-enabled.
+The default is **no source write-back**. `SDKBackend` always calls
+`AgentOptimizer.optimize(update_source=False)` so the optimizer can never commit
+early. `--update-source` is handled only by the shared pipeline: after all
+candidate evaluations, full gate acceptance, input-integrity checks, and audit
+preparation, it transactionally writes the selected prompt bundle. This rule is
+the same in fake and SDK modes. The report records the requested action and the
+terminal `writeback` state.
 
 ## Candidate Behavior
 
@@ -139,49 +151,57 @@ The fake optimizer proposes exactly two candidates:
   requested; it improves validation without protected-case regression, so the
   gate may accept it.
 
-The fake model is driven by `EvalCase.expectation`, `tags`, `protected`, and
-optional `simulated_outputs`; it does not depend on sample `case_id` names.
+The fake model reads only the prompt and user input. It cannot inspect
+expectations, labels, tags, protected markers, or case ids. Those fields are
+available only to the fake judge, failure attribution, and acceptance gate.
 
 ## Reports
 
-`optimization_report.json` includes:
+`optimization_report.json` uses `eval_optimize_loop.v2` and includes:
 
 - `schema_version`;
 - `run` metadata: mode, fake flags, trace flag, case counts, update-source flag,
   and input paths;
 - `baseline` plus compatibility fields `baseline_train` and
   `baseline_validation`;
-- all candidate train/validation results, rationale, and prompt diff;
+- `baseline_prompts`, all candidate prompt bundles, complete train/validation
+  results, rationale, and prompt diff;
+- optimizer `rounds`, `cost_summary`, and terminal `writeback`;
 - per-case deltas with `delta_type` (`new_pass`, `new_fail`, `score_up`,
   `score_down`, `unchanged`);
 - failure attribution summary and attribution accuracy when expected labels are
   present;
 - gate decisions with overfit detection, protected regressions, new hard
-  failures, excessive drops, cost fields, and SDK `not_applied_checks` when
-  per-case validation details are not exposed;
-- audit data: seed, duration, config hash, input hashes, candidate prompt hashes,
-  cost, prompt diffs, and reproducibility command.
+  failures, excessive drops, and cost fields in both fake and SDK modes;
+- audit data: seed, real duration, redacted config and raw-file hashes, input
+  hashes, candidate artifact mapping, complete/incomplete cost semantics,
+  prompt integrity journal, and reproducibility command.
 
-`gate.max_total_cost` is interpreted as the total evaluated run cost at the time
-each candidate is judged: baseline cost plus all candidates evaluated so far,
-including rejected candidates. This makes budget decisions deterministic and
-auditable when multiple candidates are considered.
+When `CostSummary.complete` is true, `gate.max_total_cost` is interpreted as the
+total evaluated run cost when each candidate is judged: baseline cost plus all
+candidates evaluated so far, including rejected candidates. When it is false,
+the report separates `known_run_cost` and `reported_optimizer_cost`, and any
+numeric total-cost gate rejects with `cost_unavailable`.
 
 `optimization_report.md` includes final decision, gate reasons, score table,
 per-case delta table, failure attribution summary, cost/audit details, prompt
 diffs, and the reproducibility command.
 
-`report.py` also writes audit artifacts under `output_dir/runs/<run_id>/`:
-
-- `config.snapshot.json`;
-- `input_hashes.json` with train, validation, optimizer, prompt,
-  `target_prompts.<field>`, and optional `gate_config` hashes;
-- fake mode: `candidate_prompts/<candidate_id>/system_prompt.txt`;
-- SDK mode: `candidate_prompts/<candidate_id>/<field_name>.txt` for every
-  returned `best_prompts` field, plus `bundle.txt` with the combined prompt
-  shown in the wrapper report;
-- `case_results/<candidate_id>_<split>.json`;
-- `prompt_diffs/<candidate_id>.diff`.
+Audit data is first written under `output_dir/runs/.<run_id>.tmp/`. After every
+final artifact and prompt-integrity check succeeds, the directory is published
+once to `output_dir/runs/<run_id>/` with no-replace semantics. Windows uses
+`MoveFileExW` without a replace flag; Linux uses
+`renameat2(RENAME_NOREPLACE)`. A POSIX platform without that atomic primitive
+fails closed. Published run directories are authoritative and immutable; the
+top-level JSON/Markdown files are convenience copies created afterward.
+
+Each run contains `pre_write_report.*`, final reports, a redacted config
+snapshot, input hashes, `baseline_prompts/`, every candidate prompt field and
+diff, baseline and candidate split results, per-round JSON, per-case deltas,
+gate decisions, evaluation failures, `writeback.json`, and the durable
+writeback journal. SDK optimizer artifacts are kept beneath `optimizer/` with a
+redacted config snapshot. `artifact_manifest.json` maps the logical artifacts
+and records the SHA-256 and byte size of every file.
 
 The repository keeps only stable examples:
 
@@ -197,9 +217,9 @@ directories are not committed.
 python -m pytest examples/optimization/eval_optimize_loop/tests
 ```
 
-The tests cover fake hidden-sample generalization, config validation, gate
-rejection paths, protected-case behavior, failure attribution, tool/knowledge
-judge paths, SDK adapter wiring through monkeypatching, deterministic report
-generation, and both CLI forms. CI can run fake mode plus the monkeypatched SDK
-smoke tests without real API credentials; real SDK/model calls are opt-in local
-or integration runs.
+The tests cover fake hidden-sample generalization, SDK EvalSet compatibility,
+config validation, gate rejection paths, protected-case behavior, failure
+attribution, tool/knowledge judge paths, SDK adapter and post-evaluator wiring
+through monkeypatching, deterministic report generation, and both CLI forms. CI
+can run fake mode plus the monkeypatched SDK smoke tests without real API
+credentials; real SDK/model calls are opt-in local or integration runs.

examples/optimization/eval_optimize_loop/data/optimizer.json

@@ -1,20 +1,48 @@
 {
-  "seed": 91,
-  "optimizer": {
-    "name": "fake_two_candidate_optimizer",
-    "description": "Deterministically proposes one overfit candidate and one safe candidate."
-  },
-  "metrics": {
-    "case_score": "mean",
-    "failure_attribution": "rule_based"
-  },
-  "gate": {
-    "min_val_score_improvement": 0.01,
-    "allow_new_hard_fail": false,
-    "protected_case_ids": [
-      "val_protected_yes_no"
+  "evaluate": {
+    "metrics": [
+      {
+        "metric_name": "final_response_avg_score",
+        "threshold": 1.0,
+        "criterion": {
+          "final_response": {
+            "text": {
+              "match": "exact",
+              "case_insensitive": false
+            }
+          }
+        }
+      }
     ],
-    "max_score_drop_per_case": 0.0,
-    "max_total_cost": 1.0
+    "num_runs": 1
+  },
+  "optimize": {
+    "eval_case_parallelism": 1,
+    "stop": {
+      "required_metrics": "all"
+    },
+    "algorithm": {
+      "name": "gepa_reflective",
+      "seed": 91,
+      "reflection_lm": {
+        "model_name": "${TRPC_AGENT_MODEL_NAME}",
+        "base_url": "${TRPC_AGENT_BASE_URL}",
+        "api_key": "${TRPC_AGENT_API_KEY}",
+        "generation_config": {
+          "max_tokens": 2048,
+          "temperature": 0.4
+        }
+      },
+      "candidate_selection_strategy": "pareto",
+      "module_selector": "round_robin",
+      "frontier_type": "instance",
+      "reflection_minibatch_size": 3,
+      "reflection_history_top_k": 2,
+      "skip_perfect_score": false,
+      "use_merge": false,
+      "max_metric_calls": 18,
+      "score_threshold": 1.0,
+      "max_iterations_without_improvement": 3
+    }
   }
 }

examples/optimization/eval_optimize_loop/eval_loop/artifacts.py

@@ -7,7 +7,6 @@
 from pathlib import Path
 from typing import Any
 
-
 ARTIFACT_COMPONENT_RE = re.compile(r"^[A-Za-z0-9_.-]+$")
 WINDOWS_RESERVED_NAMES = {
     "CON",
@@ -23,13 +22,8 @@
 def validate_artifact_component(value: Any, *, context: str) -> str:
     """Return a portable artifact component or fail before filesystem access."""
 
-    if (
-        not isinstance(value, str)
-        or value in {"", ".", ".."}
-        or len(value) > MAX_ARTIFACT_COMPONENT_LENGTH
-        or value.endswith((".", " "))
-        or not ARTIFACT_COMPONENT_RE.fullmatch(value)
-    ):
+    if (not isinstance(value, str) or value in {"", ".", ".."} or len(value) > MAX_ARTIFACT_COMPONENT_LENGTH
+            or value.endswith((".", " ")) or not ARTIFACT_COMPONENT_RE.fullmatch(value)):
         raise ValueError(f"unsafe {context}: {value!r}")
     windows_stem = value.split(".", 1)[0].upper()
     if windows_stem in WINDOWS_RESERVED_NAMES:
@@ -57,10 +51,8 @@ def validate_distinct_file_paths(
         portable_key = str(resolved).casefold()
         previous_label = resolved_keys.get(portable_key)
         if previous_label is not None:
-            raise ValueError(
-                f"{context} must be different physical files; "
-                f"{previous_label!r} and {label!r} collide case-insensitively"
-            )
+            raise ValueError(f"{context} must be different physical files; "
+                             f"{previous_label!r} and {label!r} collide case-insensitively")
 
         try:
             metadata = path.stat()
@@ -69,16 +61,11 @@ def validate_distinct_file_paths(
         except OSError as error:
             raise ValueError(f"{context} {label!r} is unavailable: {error}") from error
 
-        physical_key = (
-            (int(metadata.st_dev), int(metadata.st_ino))
-            if metadata is not None and int(metadata.st_ino) != 0
-            else None
-        )
+        physical_key = ((int(metadata.st_dev),
+                         int(metadata.st_ino)) if metadata is not None and int(metadata.st_ino) != 0 else None)
         if physical_key is not None and physical_key in physical_keys:
-            raise ValueError(
-                f"{context} must be different physical files; "
-                f"{physical_keys[physical_key]!r} and {label!r} are aliases"
-            )
+            raise ValueError(f"{context} must be different physical files; "
+                             f"{physical_keys[physical_key]!r} and {label!r} are aliases")
 
         for observed_label, observed_path, observed_exists in observed_paths:
             if metadata is None or not observed_exists:
@@ -90,10 +77,8 @@ def validate_distinct_file_paths(
             except OSError as error:
                 raise ValueError(f"{context} {label!r} is unavailable: {error}") from error
             if same_file:
-                raise ValueError(
-                    f"{context} must be different physical files; "
-                    f"{observed_label!r} and {label!r} are aliases"
-                )
+                raise ValueError(f"{context} must be different physical files; "
+                                 f"{observed_label!r} and {label!r} are aliases")
 
         resolved_keys[portable_key] = label
         if physical_key is not None: