feat(evaluation): 完成离线模型与 Trace 回放验收链路

Author: guocfu

examples/optimization/eval_optimize_loop/DESIGN.md

@@ -0,0 +1,9 @@
+# 设计边界
+
+本示例把运行环境与替代组件分开描述。`offline`、`real`、`trace` 是三种 Pipeline 运行模式。`offline` 仍使用 SDK 的 `LlmAgent`、Runner 和独立 Session，只把 Agent 内部模型替换成 `DeterministicFakeModel`，并用确定性 Candidate Provider 代替真实优化器。它用于验证 Prompt 改变是否经过真实 Agent 编排影响输出。
+
+`real` 使用真实业务模型生成回复，并由 `AgentOptimizer` 调用真实反思模型产生 Prompt 候选；只有 Gate 接受、源 Prompt 哈希未漂移且显式启用写回时，才允许更新源文件。
+
+`trace` 直接评测预录制的 `actual_conversation`，不再次运行 Agent、Model 或 Candidate Provider。它适合复现工具轨迹和生产故障，但只能证明候选版本与轨迹的关联，不能证明 Prompt 导致了该轨迹。因此 Trace 即使获得 ACCEPT，也固定跳过源 Prompt 写回。
+
+确定性 metric 负责精确匹配等硬规则。LLM Judge 若需要，应作为带 rubric 的评测指标显式配置；本示例不提供容易混淆职责的 `use_fake_judge` 开关。

examples/optimization/eval_optimize_loop/README.md

@@ -1,4 +1,12 @@
-# Evaluation + Optimization Loop — Stage 5
+# Evaluation + Optimization Loop — Stage 6
+
+## 三种运行模式
+
+- `offline`：使用真实 SDK `LlmAgent` 和 `Runner`，Agent 内部注入 `DeterministicFakeModel`，候选由确定性 Candidate Provider 生成。无需 API Key。
+- `trace`：使用 SDK 原生 `eval_mode="trace"` 回放 baseline/candidate 的 train 与 validation 轨迹，不运行 Agent、Model 或 Candidate Provider。即使 Gate 接受，也不会写回源 Prompt。
+- `real`：业务 Agent 和反思优化模型均使用真实模型，由显式入口 `run_real_integration.py --run-real` 启用。
+
+Fake Model 是 `offline` 模式中的模型实现，不是独立运行模式；Fake Candidate Provider 只替代候选生成，不替代业务 Agent。精确匹配等硬规则由 `optimizer.json` 的确定性 metric 执行。需要语义判断时，应显式配置带 rubric 的 LLM Judge，而不是使用布尔 Fake Judge 开关。详细边界见 [DESIGN.md](DESIGN.md)。
 
 This example provides an auditable evaluation and prompt-optimization loop.
 Stages 1–3 prepare an isolated prompt workspace, run baseline and candidate
@@ -8,17 +16,25 @@ common Candidate Provider boundary, an `AgentOptimizer` adapter, and guarded
 source-prompt writeback. Stage 5 publishes a complete JSON/Markdown report and
 artifact index, or preserves a standalone failure report when a run fails.
 
-The deterministic fake mode still runs without a model, API key, judge, or
-optimizer. Its built-in scenarios produce ACCEPT for `improve` and REJECT for
+The deterministic offline mode runs through the SDK Agent without an API key,
+judge, or real optimizer. Its built-in scenarios produce ACCEPT for `improve` and REJECT for
 both `no_improvement` and `overfit`:
 
 ```bash
 python examples/optimization/eval_optimize_loop/run_pipeline.py \
-  --run-id local_stage5 \
+  --run-id local_stage6 \
   --scenario improve
 ```
 
-The offline CLI remains self-contained and supports fake mode only. For an
+The same CLI also supports trace replay:
+
+```bash
+python examples/optimization/eval_optimize_loop/run_pipeline.py \
+  --config examples/optimization/eval_optimize_loop/pipeline.trace.json \
+  --scenario overfit
+```
+
+For an
 explicit real integration smoke run, configure the OpenAI-compatible business
 model connection in the environment:
 
@@ -128,7 +144,7 @@ usage remain `unavailable` because business-agent calls may not expose complete
 telemetry. In real mode, optimizer rounds, reflection calls, duration, cost, and
 token usage are reported independently from the native optimizer result; an
 unreliable or incomplete field stays `unavailable` instead of being treated as
-zero. In fake mode, optimizer-only fields are `not_applicable`.
+zero. In offline mode, optimizer-only fields are `not_applicable`.
 
 Run the Stage 1–5 tests with:
 

examples/optimization/eval_optimize_loop/analysis.py

@@ -11,7 +11,7 @@
 from .case_diff import compare_evaluations
 from .evaluation_adapter import standardize_snapshot
 from .schemas import EvaluationAnalysis
-from .schemas import FakeEvaluationSnapshot
+from .schemas import EvaluationSnapshot
 from .schemas import ObservableValue
 from .schemas import OverfitStatus
 
@@ -39,10 +39,10 @@ def _overfit_status(
 
 def build_evaluation_analysis(
     *,
-    baseline_train: FakeEvaluationSnapshot,
-    baseline_validation: FakeEvaluationSnapshot,
-    candidate_train: FakeEvaluationSnapshot,
-    candidate_validation: FakeEvaluationSnapshot,
+    baseline_train: EvaluationSnapshot,
+    baseline_validation: EvaluationSnapshot,
+    candidate_train: EvaluationSnapshot,
+    candidate_validation: EvaluationSnapshot,
     hard_case_ids: set[str],
     critical_case_ids: set[str],
     severe_case_score_drop: float,

examples/optimization/eval_optimize_loop/artifact_writer.py

@@ -23,6 +23,7 @@
 from .report_builder import render_optimization_markdown
 from .schemas import ArtifactIndex, ArtifactReference, FailureReport
 from .schemas import OptimizationReport, ReportPhase
+from .schemas import TraceCandidateProposal
 
 
 ArtifactType: TypeAlias = Literal[
@@ -536,7 +537,7 @@ def publish_report_bundle(
                 )
             )
 
-        input_specs = (
+        input_specs = [
             (
                 "input.pipeline_config",
                 Path(report.input_snapshot.pipeline_config_path),
@@ -565,7 +566,32 @@ def publish_report_bundle(
                 "validation_evalset.json",
                 "baseline_validation",
             ),
-        )
+        ]
+        if (
+            isinstance(report.candidate, TraceCandidateProposal)
+            and report.input_snapshot.trace_inputs is not None
+        ):
+            trace = report.input_snapshot.trace_inputs.scenarios[
+                report.candidate.scenario
+            ]
+            input_specs.extend(
+                [
+                    (
+                        "input.trace.candidate_train",
+                        Path(trace.train_evalset_path),
+                        trace.train_evalset_sha256,
+                        "candidate_train_trace.json",
+                        "candidate_train",
+                    ),
+                    (
+                        "input.trace.candidate_validation",
+                        Path(trace.validation_evalset_path),
+                        trace.validation_evalset_sha256,
+                        "candidate_validation_trace.json",
+                        "candidate_validation",
+                    ),
+                ]
+            )
         for artifact_id, source, expected_hash, destination_name, produced_by in input_specs:
             if copy_input_files:
                 content_validator = (

examples/optimization/eval_optimize_loop/business_agent.py

@@ -0,0 +1,102 @@
+# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
+#
+# Copyright (C) 2026 Tencent. All rights reserved.
+#
+# tRPC-Agent-Python is licensed under the Apache License, Version 2.0.
+"""共享的 SDK 业务 Agent：模型可注入，Prompt 每次重新读取。"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from uuid import uuid4
+
+from trpc_agent_sdk.agents import LlmAgent
+from trpc_agent_sdk.evaluation import TargetPrompt
+from trpc_agent_sdk.models import LLMModel
+from trpc_agent_sdk.runners import Runner
+from trpc_agent_sdk.sessions import InMemorySessionService
+from trpc_agent_sdk.types import Content
+from trpc_agent_sdk.types import GenerateContentConfig
+from trpc_agent_sdk.types import Part
+
+
+ModelFactory = Callable[[], LLMModel]
+
+
+def render_instruction(prompts: dict[str, str]) -> str:
+    """稳定拼接一个或多个工作 Prompt 字段。"""
+    if len(prompts) == 1:
+        return next(iter(prompts.values()))
+    return "\n\n".join(
+        f"## {name}\n{content}" for name, content in prompts.items()
+    )
+
+
+class BusinessAgent:
+    """以独立 SDK Session 执行一次 Prompt 敏感的业务请求。"""
+
+    def __init__(
+        self,
+        target_prompt: TargetPrompt,
+        model_factory: ModelFactory,
+        *,
+        agent_name: str,
+        app_name: str,
+        user_id: str,
+    ) -> None:
+        self._target_prompt = target_prompt
+        self._model_factory = model_factory
+        self._agent_name = agent_name
+        self._app_name = app_name
+        self._user_id = user_id
+
+    async def call_agent(self, query: str) -> str:
+        """重新读取 Prompt，运行独立 Session，并返回最终可见文本。"""
+        if not isinstance(query, str):
+            raise TypeError("query must be a string")
+
+        prompts = await self._target_prompt.read_all()
+        model = self._model_factory()
+        root_agent = LlmAgent(
+            name=self._agent_name,
+            description="Evaluation and prompt optimization business agent.",
+            model=model,
+            instruction=render_instruction(prompts),
+            generate_content_config=GenerateContentConfig(
+                temperature=0.0,
+                max_output_tokens=512,
+            ),
+        )
+        session_service = InMemorySessionService()
+        runner = Runner(
+            app_name=self._app_name,
+            agent=root_agent,
+            session_service=session_service,
+        )
+        session_id = str(uuid4())
+        await session_service.create_session(
+            app_name=self._app_name,
+            user_id=self._user_id,
+            session_id=session_id,
+            state={},
+        )
+        message = Content(
+            role="user",
+            parts=[Part.from_text(text=query)],
+        )
+        final_text = ""
+        async for event in runner.run_async(
+            user_id=self._user_id,
+            session_id=session_id,
+            new_message=message,
+        ):
+            if (
+                not event.is_final_response()
+                or not event.content
+                or not event.content.parts
+            ):
+                continue
+            for part in event.content.parts:
+                if not part.thought and part.text:
+                    final_text += part.text
+        return final_text.strip()

examples/optimization/eval_optimize_loop/candidate_provider.py

@@ -21,7 +21,7 @@
 
 from .fake.candidate_provider import DeterministicFakeCandidateProvider
 from .schemas import CandidateProposal
-from .schemas import FakeCandidateScenario
+from .schemas import CandidateScenario
 from .schemas import OptimizerCandidateProposal
 from .schemas import OptimizerRuntimeParameters
 
@@ -75,7 +75,7 @@ async def propose(self, request: CandidateRequest) -> CandidateGeneration:
 class FakeCandidateProviderAdapter:
     """Lift the pure synchronous fake provider into the common async boundary."""
 
-    def __init__(self, scenario: FakeCandidateScenario) -> None:
+    def __init__(self, scenario: CandidateScenario) -> None:
         self._scenario = scenario
 
     async def propose(self, request: CandidateRequest) -> CandidateGeneration:

examples/optimization/eval_optimize_loop/config.py

@@ -30,11 +30,29 @@
 
 
 class ExecutionConfig(EvalBaseModel):
-    """How a later phase obtains an optimization candidate."""
+    """Pipeline execution mode and deterministic candidate scenario."""
 
-    mode: Literal["fake", "real", "trace"] = "fake"
-    fake_candidate_scenario: Literal["improve", "no_improvement", "overfit"] = "improve"
-    use_fake_judge: bool = False
+    mode: Literal["offline", "real", "trace"] = "offline"
+    candidate_scenario: Literal["improve", "no_improvement", "overfit"] = "improve"
+
+    @model_validator(mode="before")
+    @classmethod
+    def _reject_removed_execution_options(cls, value: object) -> object:
+        if not isinstance(value, dict):
+            return value
+        if value.get("mode") == "fake":
+            raise ValueError("execution.mode='fake' was renamed to 'offline'")
+        if "use_fake_judge" in value:
+            raise ValueError(
+                "execution.use_fake_judge was removed; configure evaluation "
+                "metrics or rubric explicitly in optimizer.json"
+            )
+        if "fake_candidate_scenario" in value:
+            raise ValueError(
+                "execution.fake_candidate_scenario was renamed to "
+                "execution.candidate_scenario"
+            )
+        return value
 
 
 class InputPathsConfig(EvalBaseModel):
@@ -72,6 +90,37 @@ def _require_non_empty_relative_path(cls, value: str) -> str:
         return value
 
 
+class TraceCandidateInputsConfig(EvalBaseModel):
+    """一个 Trace 候选版本的评测集和 Prompt 快照路径。"""
+
+    train_evalset: str
+    validation_evalset: str
+    prompts: list[PromptFieldConfig] = Field(min_length=1)
+
+    @field_validator("train_evalset", "validation_evalset")
+    @classmethod
+    def _require_relative_trace_path(cls, value: str) -> str:
+        if not value.strip() or Path(value).is_absolute():
+            raise ValueError("trace evalset path must be a non-empty relative path")
+        return value
+
+
+class TraceInputsConfig(EvalBaseModel):
+    """三个确定性候选场景的 Trace 输入。"""
+
+    candidates: dict[
+        Literal["improve", "no_improvement", "overfit"],
+        TraceCandidateInputsConfig,
+    ]
+
+    @model_validator(mode="after")
+    def _require_all_scenarios(self) -> "TraceInputsConfig":
+        required = {"improve", "no_improvement", "overfit"}
+        if set(self.candidates) != required:
+            raise ValueError("trace_inputs must define improve, no_improvement, and overfit")
+        return self
+
+
 class RunConfig(EvalBaseModel):
     """Reproducibility and workspace location settings."""
 
@@ -186,12 +235,31 @@ class PipelineConfig(EvalBaseModel):
     reporting: ReportingConfig = Field(default_factory=ReportingConfig)
     artifacts: ArtifactConfig = Field(default_factory=ArtifactConfig)
     writeback: WritebackConfig = Field(default_factory=WritebackConfig)
+    trace_inputs: Optional[TraceInputsConfig] = None
 
     @model_validator(mode="after")
     def _require_unique_prompt_names(self) -> "PipelineConfig":
         names = [prompt.name for prompt in self.prompts]
         if len(names) != len(set(names)):
             raise ValueError("prompts must not contain duplicate field names")
+        if self.execution.mode == "trace":
+            if self.trace_inputs is None:
+                raise ValueError("trace mode requires trace_inputs")
+            if self.writeback.enabled:
+                raise ValueError("trace mode does not allow source Prompt writeback")
+            expected = set(names)
+            for scenario, inputs in self.trace_inputs.candidates.items():
+                candidate_names = [prompt.name for prompt in inputs.prompts]
+                if len(candidate_names) != len(set(candidate_names)):
+                    raise ValueError(
+                        f"trace candidate {scenario} has duplicate prompt names"
+                    )
+                if set(candidate_names) != expected:
+                    raise ValueError(
+                        f"trace candidate {scenario} prompt fields must match baseline"
+                    )
+        elif self.trace_inputs is not None:
+            raise ValueError("trace_inputs is only allowed in trace mode")
         return self
 
 

examples/optimization/eval_optimize_loop/data/traces/baseline.train.evalset.json

@@ -0,0 +1 @@
+{"eval_set_id":"trace_train","eval_cases":[{"eval_id":"train_output_format","eval_mode":"trace","conversation":[{"invocation_id":"train_output_format","user_content":{"role":"user","parts":[{"text":"How can I update my email address?"}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"account\",\"message\":\"Open profile settings to update your email.\"}"}]}}],"actual_conversation":[{"invocation_id":"train_output_format","user_content":{"role":"user","parts":[{"text":"How can I update my email address?"}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"account\",\"message\":\"Open profile settings to update your email.\"}"}]}}]},{"eval_id":"train_tool_choice","eval_mode":"trace","conversation":[{"invocation_id":"train_tool_choice","user_content":{"role":"user","parts":[{"text":"Check the status of order A100."}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"order_lookup\",\"message\":\"Checking order A100.\"}"}]}}],"actual_conversation":[{"invocation_id":"train_tool_choice","user_content":{"role":"user","parts":[{"text":"Check the status of order A100."}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"general_support\",\"message\":\"Please provide more details so I can route your request.\"}"}]}}]},{"eval_id":"train_tool_arguments","eval_mode":"trace","conversation":[{"invocation_id":"train_tool_arguments","user_content":{"role":"user","parts":[{"text":"Look up order B-204 for customer 17."}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"order_lookup\",\"message\":\"Checking order B-204 for customer 17.\"}"}]}}],"actual_conversation":[{"invocation_id":"train_tool_arguments","user_content":{"role":"user","parts":[{"text":"Look up order B-204 for customer 17."}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"general_support\",\"message\":\"Please provide more details so I can route your request.\"}"}]}}]}]}

examples/optimization/eval_optimize_loop/data/traces/baseline.validation.evalset.json

@@ -0,0 +1 @@
+{"eval_set_id":"trace_validation","eval_cases":[{"eval_id":"val_paraphrase","eval_mode":"trace","conversation":[{"invocation_id":"val_paraphrase","user_content":{"role":"user","parts":[{"text":"Where do I change the address tied to my account?"}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"account\",\"message\":\"Open profile settings to update your address.\"}"}]}}],"actual_conversation":[{"invocation_id":"val_paraphrase","user_content":{"role":"user","parts":[{"text":"Where do I change the address tied to my account?"}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"general_support\",\"message\":\"Please provide more details so I can route your request.\"}"}]}}]},{"eval_id":"val_knowledge_recall","eval_mode":"trace","conversation":[{"invocation_id":"val_knowledge_recall","user_content":{"role":"user","parts":[{"text":"How long does standard shipping usually take?"}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"shipping_policy\",\"message\":\"Standard shipping normally takes 3-5 business days.\"}"}]}}],"actual_conversation":[{"invocation_id":"val_knowledge_recall","user_content":{"role":"user","parts":[{"text":"How long does standard shipping usually take?"}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"general_support\",\"message\":\"Please provide more details so I can route your request.\"}"}]}}]},{"eval_id":"val_refund_route","eval_mode":"trace","conversation":[{"invocation_id":"val_refund_route","user_content":{"role":"user","parts":[{"text":"I was charged twice and need the duplicate payment refunded."}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"billing_refund\",\"message\":\"I will route this duplicate charge for refund review.\"}"}]}}],"actual_conversation":[{"invocation_id":"val_refund_route","user_content":{"role":"user","parts":[{"text":"I was charged twice and need the duplicate payment refunded."}]},"final_response":{"role":"model","parts":[{"text":"{\"route\":\"billing_refund\",\"message\":\"I will route this duplicate charge for refund review.\"}"}]}}]}]}