feat(evaluation): 完成第五阶段报告与审计闭环

Author: guocfu

examples/optimization/eval_optimize_loop/README.md

@@ -1,19 +1,20 @@
-# Evaluation + Optimization Loop — Stage 4
+# Evaluation + Optimization Loop — Stage 5
 
 This example provides an auditable evaluation and prompt-optimization loop.
 Stages 1–3 prepare an isolated prompt workspace, run baseline and candidate
 evaluations on both train and validation datasets, normalize failures, build
 case diffs, detect overfitting, and apply an independent Gate. Stage 4 adds a
 common Candidate Provider boundary, an `AgentOptimizer` adapter, and guarded
-source-prompt writeback.
+source-prompt writeback. Stage 5 publishes a complete JSON/Markdown report and
+artifact index, or preserves a standalone failure report when a run fails.
 
 The deterministic fake mode still runs without a model, API key, judge, or
 optimizer. Its built-in scenarios produce ACCEPT for `improve` and REJECT for
 both `no_improvement` and `overfit`:
 
 ```bash
 python examples/optimization/eval_optimize_loop/run_pipeline.py \
-  --run-id local_stage4 \
+  --run-id local_stage5 \
   --scenario improve
 ```
 
@@ -38,7 +39,8 @@ credential reuse the environment without writing their resolved values to run
 artifacts. `--run-real` is mandatory so an accidental command cannot spend API
 quota. This entry always uses `pipeline.real.json`, where source writeback is
 disabled, and reports ACCEPT or REJECT without treating REJECT as a process
-failure.
+failure. Both CLIs print the paths of the JSON report, Markdown report, and
+artifact index after a completed run.
 
 Applications with a custom agent can still use the Python integration point.
 Set `execution.mode` to `real`, prepare the run, and inject an async
@@ -67,6 +69,45 @@ pipeline evaluations: baseline train/validation and candidate train/validation.
 scores, and configuration snapshot are retained under `runs/<run-id>/optimizer/`
 when `artifacts.retain_optimizer_native_artifacts` is enabled.
 
+## Report artifacts
+
+A completed run atomically publishes the formal bundle at
+`runs/<run-id>/report/`:
+
+```text
+report/
+├── optimization_report.json
+├── optimization_report.md
+├── artifact_index.json
+├── inputs/
+│   ├── pipeline_config.json
+│   ├── optimizer_config.json
+│   ├── train_evalset.json
+│   └── validation_evalset.json
+├── evaluations/
+│   ├── baseline_train.json
+│   ├── baseline_validation.json
+│   ├── candidate_train.json
+│   └── candidate_validation.json
+└── prompts/
+    ├── baseline/
+    └── candidate/
+```
+
+The formal `report/` directory is visible only after all required artifacts
+have been written and validated. If any post-preparation phase fails, the
+pipeline does not leave a partial formal report; it atomically writes
+`runs/<run-id>/failure_report.json` with the failed phase, completed phases,
+sanitized error information, source Prompt hashes, and already existing
+artifacts. A failure-report write error is surfaced together with the original
+pipeline error.
+
+`artifact_index.json` records each artifact's relative path, SHA-256 hash, byte
+size, producer phase, and availability. Consumers can use those fields to
+verify that an artifact has not drifted since publication. Input copies are
+validated against the preparation snapshot, and sensitive resolved credentials
+are not accepted into the report bundle.
+
 Source prompts are updated only when all of these conditions hold:
 
 - Gate returns ACCEPT;
@@ -81,13 +122,15 @@ returned as failed. If rollback integrity cannot be proven, the pipeline raises
 an error instead of claiming the source is safe. The checked-in configuration
 keeps writeback disabled by default.
 
-Elapsed duration is observable. Full monetary cost and token usage remain
-`unavailable` because an injected business agent may make calls that the SDK
-optimizer does not account for; native optimizer resource fields are retained
-without treating missing values as zero. JSON/Markdown reports and an artifact
-index are Stage 5 work.
+The report separates whole-pipeline resources from optimizer-only observations.
+Pipeline duration is observable, while full pipeline monetary cost and token
+usage remain `unavailable` because business-agent calls may not expose complete
+telemetry. In real mode, optimizer rounds, reflection calls, duration, cost, and
+token usage are reported independently from the native optimizer result; an
+unreliable or incomplete field stays `unavailable` instead of being treated as
+zero. In fake mode, optimizer-only fields are `not_applicable`.
 
-Run the Stage 1–4 tests with:
+Run the Stage 1–5 tests with:
 
 ```bash
 .venv/bin/pytest -q \
@@ -96,5 +139,8 @@ Run the Stage 1–4 tests with:
   tests/evaluation/test_eval_optimize_loop_stage3a.py \
   tests/evaluation/test_eval_optimize_loop_stage3b.py \
   tests/evaluation/test_eval_optimize_loop_stage4.py \
-  tests/evaluation/test_eval_optimize_loop_real_integration.py
+  tests/evaluation/test_eval_optimize_loop_real_integration.py \
+  tests/evaluation/test_eval_optimize_loop_stage5_report_builder.py \
+  tests/evaluation/test_eval_optimize_loop_stage5_artifacts.py \
+  tests/evaluation/test_eval_optimize_loop_stage5_pipeline.py
 ```

examples/optimization/eval_optimize_loop/artifact_writer.py

@@ -54,6 +54,21 @@
     "token",
     "xapikey",
 }
+_SENSITIVE_CONFIG_KEY_SUFFIXES = {
+    "accesstoken",
+    "apikey",
+    "authtoken",
+    "baseurl",
+    "bearertoken",
+    "clientsecret",
+    "credential",
+    "credentials",
+    "endpointurl",
+    "password",
+    "passwd",
+    "privatekey",
+    "secretkey",
+}
 _APPROVED_SENSITIVE_VALUES = {
     "",
     "${TRPC_AGENT_API_KEY}",
@@ -167,7 +182,22 @@ def _normalized_sensitive_key(key: str) -> str:
     return key.replace("_", "").replace("-", "").casefold()
 
 
+def _is_sensitive_config_key(key: str) -> bool:
+    normalized = _normalized_sensitive_key(key)
+    return normalized in _SENSITIVE_CONFIG_KEYS or any(
+        normalized.endswith(suffix)
+        for suffix in _SENSITIVE_CONFIG_KEY_SUFFIXES
+    )
+
+
 def _validate_sensitive_config_values(value: object, *, path: str = "$") -> None:
+    if isinstance(value, str):
+        if value.strip().casefold().startswith(("http://", "https://")):
+            raise ArtifactWriteError(
+                "sensitive optimizer config value is not an approved "
+                f"placeholder: {path}"
+            )
+        return
     if isinstance(value, list):
         for index, item in enumerate(value):
             _validate_sensitive_config_values(item, path=f"{path}[{index}]")
@@ -176,7 +206,7 @@ def _validate_sensitive_config_values(value: object, *, path: str = "$") -> None
         return
     for key, item in value.items():
         item_path = f"{path}.{key}"
-        if _normalized_sensitive_key(key) in _SENSITIVE_CONFIG_KEYS:
+        if _is_sensitive_config_key(key):
             if not isinstance(item, str) or item not in _APPROVED_SENSITIVE_VALUES:
                 raise ArtifactWriteError(
                     "sensitive optimizer config value is not an approved "