docs(examples): clarify counterfactual attribution boundary

Author: Leoluis0705

examples/optimization/counterfactual_trace_loop/DESIGN.md

@@ -8,6 +8,10 @@
 
 ## 技术附录
 
+### 与现有方案的边界
+
+PR #159 主要依据失败文本和预期/实际轨迹的静态差异归因；PR #161 重点在优化候选回放、bootstrap 与 Pareto 决策。本方案不把静态差异直接视为原因，而是把它转化为局部反事实干预，再由同一个 `AgentEvaluator` 重评。只有可重复观察到 metric 修复的干预才形成归因证据，且该机制同时用于 baseline 失败和 candidate 新增退化。
+
 - 反事实归因：结论来自真实 metric delta，不依赖 case ID、failure reason 或人工标签。
 - Prompt actionability：只有 agent behavior failure 能够选择优化表面。
 - Gate：关键检查必须全部通过，证据不足时以 `NEEDS_REVIEW` 拒绝。

examples/optimization/counterfactual_trace_loop/README.md

@@ -2,14 +2,20 @@
 
 This example closes evaluation and prompt optimization with evidence from trace interventions. Unlike loops that classify a failure from reason text or sample metadata, it deep-copies the actual trace, changes one execution surface, and sends the counterfactual `EvalCase` through the same public `AgentEvaluator` metrics. Evaluation-data, evaluator, and infrastructure failures are excluded from prompt optimization.
 
+## Distinction from other Issue #91 proposals
+
+PR #159 uses rule-first attribution from failure text and static expected/actual trace differences. PR #161 replays optimizer candidates and combines static trace/rubric attribution with bootstrap and Pareto selection. This example addresses a different question: which minimal trace intervention causally repairs the failing metric when the same `AgentEvaluator` evaluates it again?
+
+The attribution evidence is therefore an observed before/after metric delta from `replace_final_response`, `replace_tool_name`, `replace_tool_arguments`, or a bounded combination. Static trace differences can propose an intervention, but cannot by themselves establish the diagnosis. The same mechanism is reused after candidate validation to locate regressions. No case ID, expected failure label, attribution hint, or hand-authored metric score participates in the decision.
+
 ## Quick start
 
 ```bash
-python examples/optimization/eval_optimize_loop/run_counterfactual_probe.py
-python examples/optimization/eval_optimize_loop/run_pipeline.py --mode fake
-python examples/optimization/eval_optimize_loop/run_pipeline.py --mode trace
-python examples/optimization/eval_optimize_loop/run_pipeline.py --mode fake --candidate-profile accepted
-python examples/optimization/eval_optimize_loop/run_pipeline.py --mode fake --candidate-profile ineffective
+python examples/optimization/counterfactual_trace_loop/run_counterfactual_probe.py
+python examples/optimization/counterfactual_trace_loop/run_pipeline.py --mode fake
+python examples/optimization/counterfactual_trace_loop/run_pipeline.py --mode trace
+python examples/optimization/counterfactual_trace_loop/run_pipeline.py --mode fake --candidate-profile accepted
+python examples/optimization/counterfactual_trace_loop/run_pipeline.py --mode fake --candidate-profile ineffective
 ```
 
 Both fake and trace modes need no API key. Trace mode replays `actual_conversation`; `conversation` remains the expected trace. The fake optimizer is prompt-sensitive and introduces an intentionally broad billing rule without reading case IDs.

examples/optimization/counterfactual_trace_loop/pipeline/pipeline.py

@@ -366,7 +366,7 @@ async def run_pipeline(
             "prompt_hashes": {k: _sha(v) for k, v in prompt_paths.items()},
             "write_back": write_back,
             "reproduction_command": (
-                "python examples/optimization/eval_optimize_loop/run_pipeline.py "
+                "python examples/optimization/counterfactual_trace_loop/run_pipeline.py "
                 f"--mode {mode} --candidate-profile {candidate_profile}"
             ),
         },

examples/optimization/counterfactual_trace_loop/run_counterfactual_probe.py

@@ -12,7 +12,7 @@
 if str(REPO_ROOT) not in sys.path:
     sys.path.insert(0, str(REPO_ROOT))
 
-from examples.optimization.eval_optimize_loop.pipeline.probe import (  # noqa: E402
+from examples.optimization.counterfactual_trace_loop.pipeline.probe import (  # noqa: E402
     run_counterfactual_probe,
 )
 

examples/optimization/counterfactual_trace_loop/run_pipeline.py

@@ -11,7 +11,7 @@
 ROOT = HERE.parents[2]
 if str(ROOT) not in sys.path:
     sys.path.insert(0, str(ROOT))
-from examples.optimization.eval_optimize_loop.pipeline.pipeline import run_pipeline  # noqa: E402
+from examples.optimization.counterfactual_trace_loop.pipeline.pipeline import run_pipeline  # noqa: E402
 
 
 def main():

examples/optimization/counterfactual_trace_loop/sample_output/optimization_report.json

@@ -947,7 +947,7 @@
       {
         "name": "latency_budget",
         "passed": true,
-        "observed": 0.29256009999880916,
+        "observed": 0.3700505000015255,
         "threshold": 180,
         "reason": "passed"
       },
@@ -970,7 +970,7 @@
   "audit": {
     "seed": 42,
     "mode": "trace",
-    "duration_seconds": 0.29256009999880916,
+    "duration_seconds": 0.3700505000015255,
     "cost": {
       "total": 0.0
     },
@@ -997,7 +997,7 @@
         "system_prompt": "4e1c6a4ef268a492679ad6888230217d874733817e27137644f3364038642be9"
       }
     },
-    "reproduction_command": "python examples/optimization/eval_optimize_loop/run_pipeline.py --mode trace --candidate-profile overfit"
+    "reproduction_command": "python examples/optimization/counterfactual_trace_loop/run_pipeline.py --mode trace --candidate-profile overfit"
   },
   "known_limitations": [
     "A local trace edit can be structurally valid but semantically incoherent with an original tool response.",

tests/evaluation/test_counterfactual_trace_probe.py

@@ -14,11 +14,11 @@
 
 from trpc_agent_sdk.types import FunctionResponse
 
-from examples.optimization.eval_optimize_loop.pipeline.interventions import (
+from examples.optimization.counterfactual_trace_loop.pipeline.interventions import (
     InterventionKind,
     build_counterfactual,
 )
-from examples.optimization.eval_optimize_loop.pipeline.probe import (
+from examples.optimization.counterfactual_trace_loop.pipeline.probe import (
     build_probe_cases,
     evaluate_trace_cases,
     run_counterfactual_probe,

tests/evaluation/test_eval_optimize_loop.py

@@ -8,24 +8,24 @@
 
 import pytest
 
-from examples.optimization.eval_optimize_loop.pipeline.diagnosis import (
+from examples.optimization.counterfactual_trace_loop.pipeline.diagnosis import (
     InfrastructureFailure,
     attribute_from_evidence,
     build_failure_digest,
     classify_non_agent_failure,
     select_target_prompts,
 )
-from examples.optimization.eval_optimize_loop.pipeline.gate import evaluate_gate
-from examples.optimization.eval_optimize_loop.pipeline.input_audit import audit_eval_cases
-from examples.optimization.eval_optimize_loop.pipeline.models import CounterfactualEvidence
-from examples.optimization.eval_optimize_loop.fake.model import generate_trace
-from examples.optimization.eval_optimize_loop.pipeline.optimizer import (
+from examples.optimization.counterfactual_trace_loop.pipeline.gate import evaluate_gate
+from examples.optimization.counterfactual_trace_loop.pipeline.input_audit import audit_eval_cases
+from examples.optimization.counterfactual_trace_loop.pipeline.models import CounterfactualEvidence
+from examples.optimization.counterfactual_trace_loop.fake.model import generate_trace
+from examples.optimization.counterfactual_trace_loop.pipeline.optimizer import (
     apply_if_accepted,
     run_real_optimizer,
 )
-from examples.optimization.eval_optimize_loop.pipeline.pipeline import run_pipeline
-from examples.optimization.eval_optimize_loop.pipeline.pipeline import _evaluate_with_triage
-from examples.optimization.eval_optimize_loop.pipeline.probe import build_probe_cases
+from examples.optimization.counterfactual_trace_loop.pipeline.pipeline import run_pipeline
+from examples.optimization.counterfactual_trace_loop.pipeline.pipeline import _evaluate_with_triage
+from examples.optimization.counterfactual_trace_loop.pipeline.probe import build_probe_cases
 
 
 def _evidence(name: str, passed: bool, repaired: list[str]) -> CounterfactualEvidence:
@@ -289,7 +289,7 @@ async def test_fake_and_trace_pipeline_have_same_schema_without_api_key(tmp_path
     for key in list(__import__("os").environ):
         if "API_KEY" in key:
             monkeypatch.delenv(key, raising=False)
-    base = Path(__file__).parents[2] / "examples" / "optimization" / "eval_optimize_loop"
+    base = Path(__file__).parents[2] / "examples" / "optimization" / "counterfactual_trace_loop"
     fake = await run_pipeline(base, "fake", tmp_path / "fake")
     trace = await run_pipeline(base, "trace", tmp_path / "trace")
     assert fake.keys() == trace.keys()
@@ -302,7 +302,7 @@ async def test_fake_and_trace_pipeline_have_same_schema_without_api_key(tmp_path
 
 @pytest.mark.asyncio
 async def test_real_evaluations_drive_accept_overfit_and_ineffective_decisions(tmp_path):
-    base = Path(__file__).parents[2] / "examples" / "optimization" / "eval_optimize_loop"
+    base = Path(__file__).parents[2] / "examples" / "optimization" / "counterfactual_trace_loop"
     accepted = await run_pipeline(base, "fake", tmp_path / "accepted", candidate_profile="accepted")
     overfit = await run_pipeline(base, "fake", tmp_path / "overfit", candidate_profile="overfit")
     ineffective = await run_pipeline(base, "fake", tmp_path / "ineffective", candidate_profile="ineffective")
@@ -317,7 +317,7 @@ async def test_real_evaluations_drive_accept_overfit_and_ineffective_decisions(t
 
 @pytest.mark.asyncio
 async def test_real_optimizer_receives_filtered_train_and_never_updates_source(tmp_path, monkeypatch):
-    base = Path(__file__).parents[2] / "examples" / "optimization" / "eval_optimize_loop"
+    base = Path(__file__).parents[2] / "examples" / "optimization" / "counterfactual_trace_loop"
     captured = {}
 
     async def spy(**kwargs):
@@ -347,7 +347,7 @@ async def spy(**kwargs):
         )
 
     monkeypatch.setattr(
-        "examples.optimization.eval_optimize_loop.pipeline.optimizer.AgentOptimizer.optimize",
+        "examples.optimization.counterfactual_trace_loop.pipeline.optimizer.AgentOptimizer.optimize",
         spy,
     )
     result = await run_real_optimizer(
@@ -385,7 +385,7 @@ async def spy(**kwargs):
 
 @pytest.mark.asyncio
 async def test_real_optimizer_restores_prompt_if_optimizer_mutates_then_fails(tmp_path, monkeypatch):
-    base = Path(__file__).parents[2] / "examples" / "optimization" / "eval_optimize_loop"
+    base = Path(__file__).parents[2] / "examples" / "optimization" / "counterfactual_trace_loop"
     prompt = tmp_path / "router.md"
     prompt.write_text("baseline", encoding="utf-8")
 
@@ -394,7 +394,7 @@ async def broken_optimizer(**kwargs):
         raise RuntimeError("optimizer failed")
 
     monkeypatch.setattr(
-        "examples.optimization.eval_optimize_loop.pipeline.optimizer.AgentOptimizer.optimize",
+        "examples.optimization.counterfactual_trace_loop.pipeline.optimizer.AgentOptimizer.optimize",
         broken_optimizer,
     )
 
@@ -414,15 +414,15 @@ async def broken_optimizer(**kwargs):
 
 @pytest.mark.asyncio
 async def test_pipeline_rejects_unsupported_real_mode(tmp_path):
-    base = Path(__file__).parents[2] / "examples" / "optimization" / "eval_optimize_loop"
+    base = Path(__file__).parents[2] / "examples" / "optimization" / "counterfactual_trace_loop"
 
     with pytest.raises(ValueError, match="fake or trace"):
         await run_pipeline(base, "real", tmp_path)
 
 
 @pytest.mark.asyncio
 async def test_report_contains_trace_status_evidence_and_limitations(tmp_path):
-    base = Path(__file__).parents[2] / "examples" / "optimization" / "eval_optimize_loop"
+    base = Path(__file__).parents[2] / "examples" / "optimization" / "counterfactual_trace_loop"
     report = await run_pipeline(base, "trace", tmp_path)
     case = report["baseline"]["train"]["case_results"][0]
     assert {"case_id", "passed", "metrics", "failure_reason", "trace_summary"} <= case.keys()
@@ -435,7 +435,7 @@ async def test_report_contains_trace_status_evidence_and_limitations(tmp_path):
 
 @pytest.mark.asyncio
 async def test_fake_round_audit_contains_candidate_prompts_and_evaluation(tmp_path):
-    base = Path(__file__).parents[2] / "examples" / "optimization" / "eval_optimize_loop"
+    base = Path(__file__).parents[2] / "examples" / "optimization" / "counterfactual_trace_loop"
 
     report = await run_pipeline(base, "fake", tmp_path, candidate_profile="accepted")
 
@@ -450,7 +450,7 @@ async def test_fake_round_audit_contains_candidate_prompts_and_evaluation(tmp_pa
 
 
 def test_committed_sample_output_has_current_report_contract():
-    sample = Path(__file__).parents[2] / "examples" / "optimization" / "eval_optimize_loop" / "sample_output"
+    sample = Path(__file__).parents[2] / "examples" / "optimization" / "counterfactual_trace_loop" / "sample_output"
     report = json.loads((sample / "optimization_report.json").read_text(encoding="utf-8"))
     assert report["schema_version"] == "1.0"
     assert report["baseline"]["train"]["case_results"]
@@ -471,7 +471,7 @@ async def evaluator(case_batch, workspace):
         return {case.eval_id: {"tool_trajectory_avg_score": 1.0}}
 
     monkeypatch.setattr(
-        "examples.optimization.eval_optimize_loop.pipeline.pipeline.evaluate_trace_cases",
+        "examples.optimization.counterfactual_trace_loop.pipeline.pipeline.evaluate_trace_cases",
         evaluator,
     )
     metrics, failures = await _evaluate_with_triage(cases, tmp_path)