Skip to content

Commit 8f9a323

Browse files
16yunH16yunH
committed
examples: add eval optimization regression loop
Add a no-key Evaluation and Optimization pipeline that baselines train and validation data, captures every GEPA proposal, independently re-evaluates candidates, applies regression and resource gates, and emits auditable JSON and Markdown reports. Include deterministic replay data, prompt candidates, failure attribution, paired bootstrap confidence intervals, Pareto selection, data-leakage checks, sample output, and regression tests. Fixes #91 RELEASE NOTES: Add an auditable offline Evaluation and Optimization regression loop example.
1 parent e113610 commit 8f9a323

26 files changed

Lines changed: 7155 additions & 0 deletions

‎examples/optimization/eval_optimize_loop/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
runs/
2+
sample_output/optimizer/
3+
sample_output/traces/

‎examples/optimization/eval_optimize_loop/DESIGN.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# 设计说明（约 400 字）
2+
3+
闭环以 `EvalOptimizePipeline.run()` 作为唯一外部接口，把数据校验、基线评估、提示词优化、候选复评、差异计算、门禁和报告隐藏在同一深模块内。优化阶段直接复用 `AgentOptimizer`、`TargetPrompt` 与真实 GEPA；无密钥模式仅通过 `ModelRegistry` 注入离线 agent、judge 和 reflector，因而仍会经过 `LlmAgent`、内置 rubric 解析及优化器回调，不依赖 monkeypatch。外层将每个候选运行物化为 trace，再用 `AgentEvaluator` 独立评测 train 与 validation，避免采信优化器自己的聚合分数。候选集合按完整提示词哈希去重，即使优化器内部提前拒绝某次提案，外层仍会捕获并独立复评，防止遗漏验证回退。逐 case 差异区分新增通过、新增失败、升分、降分和不变；归因按工具执行、参数、格式、知识召回、rubric、最终回答的根因优先级输出证据。SDK 聚合器偶尔不保留逐 rubric 详情，格式归因依次采用详情、回放证据和请求中明确的 JSON、单行或 Markdown 约束，否定式要求不触发检查；judge 无有效 verdict 时单列 evaluation_error，避免把系统错误或内容失败误报为格式失败。Gate 对验证增益、配对 bootstrap 下界、新增 hard fail、关键 case、过拟合及资源预算做可配置 AND，并先过滤不安全候选，再计算质量、token、P95 时延 Pareto。运行前检查跨集合重复、近重复和答案泄漏；运行后保存完整 prompt、seed、哈希、耗时、成本、trace 与 JSON/Markdown 报告。默认始终恢复源 prompt，只有显式允许且 Gate 全通过才原子回写。

‎examples/optimization/eval_optimize_loop/README.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Evaluation + Optimization 自动回归闭环
2+
3+
本示例把一次 prompt 优化从“分数变高了”升级为可审计的发布决策：先在 train/validation 上建立 baseline，再让真实 `AgentOptimizer` 与 GEPA 生成候选，随后对**每个去重候选**做独立 trace 复评、逐 case diff、失败归因和 Gate 判定，最后同时输出 JSON 与 Markdown 报告。默认模式不读取任何 API Key，也不访问网络。
4+
5+
## 一键运行
6+
7+
在仓库根目录执行：
8+
9+
```bash
10+
pip install -e ".[eval,optimize]"
11+
python examples/optimization/eval_optimize_loop/run_pipeline.py
12+
```
13+
14+
无需设置 `OPENAI_API_KEY`、`TRPC_AGENT_API_KEY` 或其他模型凭据。指定输出目录：
15+
16+
```bash
17+
python examples/optimization/eval_optimize_loop/run_pipeline.py \
18+
--output-dir /tmp/eval-optimize-loop
19+
```
20+
21+
命令正常完成时输出 `decision=accepted` 与报告路径。默认不会覆盖源 prompt；只有显式传入 `--apply-if-accepted` 且所有必选 Gate 均通过，才会原子回写最终候选。
22+
23+
## 为什么不是“伪造优化”
24+
25+
离线 provider 通过框架公开的 `ModelRegistry` 注册三个模型名，并真实经过 SDK 的生产调用链：
26+
27+
```text
28+
AgentOptimizer.optimize
29+
-> GepaReflectiveOptimizer
30+
-> gepa.optimize
31+
-> AgentEvaluator
32+
-> LlmAgent(offline/agent)
33+
-> 内置 LLMRubricResponseEvaluator(offline/judge)
34+
35+
reflection
36+
-> _OptimizeModelCallable
37+
-> LlmAgent(offline/reflector)
38+
```
39+
40+
离线模型只替换远程推理这一处依赖；候选写入/恢复、GEPA 采样和筛选、rubric JSON 解析、评估聚合、回调及优化器 artifact 全部使用真实框架实现，没有 monkeypatch。外层回归会把每种 prompt 的确定性运行结果物化为 `actual_conversation`，再用 `AgentEvaluator` 的 trace-only 模式独立评分。
41+
42+
## 示例数据与三类候选
43+
44+
`data/train.evalset.json` 与 `data/val.evalset.json` 各有 3 个 case。它们不在实现代码中硬编码 case id，离线运行行为完全来自每个 case 的 `session_input.state.variant_traces`。
45+
46+
| 候选 | Train | Validation | 预期 Gate |
47+
| --- | --- | --- | --- |
48+
| `ineffective` | 无提升 | 无提升 | 拒绝：validation 增益不足 |
49+
| `overfit` | 3/3 通过 | 关键 JSON case 从通过变失败 | 拒绝：hard fail、关键 case 回退、过拟合 |
50+
| `robust` | 3/3 通过 | 3/3 通过 | 接受 |
51+
52+
六个 case 覆盖：最终回答不匹配、工具执行错误、工具参数错误、LLM rubric 失败、知识召回不足和格式失败。归因器从 actual/expected 工具轨迹、工具响应和 evaluator metric 中推导原因，不读取预埋的 `failure_category` 标签。格式归因优先使用逐 rubric 详情，其次使用 replay 证据，最后只解析请求中明确、非否定的 JSON、单行或 Markdown 约束；evaluator 没有产生有效 verdict 时单列 `evaluation_error`，不会伪装成语义失败。
53+
54+
## 输入
55+
56+
| 文件 | 用途 |
57+
| --- | --- |
58+
| `pipeline.json` | 路径清单、seed、bootstrap 次数和是否回写 |
59+
| `optimizer.json` | 可直接被 SDK `load_optimize_config()` 校验的 GEPA 配置 |
60+
| `regression.metrics.json` | 外层 trace 复评的回答、轨迹、rubric、知识召回指标 |
61+
| `gate.json` | 增益、置信下界、hard fail、关键 case、过拟合和预算规则 |
62+
| `agent/prompts/system.md` | baseline `TargetPrompt` 源文件 |
63+
| `agent/prompts/candidates/*.md` | 离线 reflection model 的确定性候选 |
64+
| `data/*.evalset.json` | train / validation case 与 replay trace |
65+
66+
## Gate 语义
67+
68+
每一项都在报告中记录 `passed / required / actual / expected / reason`。必选项做 AND：
69+
70+
1. `AgentOptimizer` 成功结束；
71+
2. validation pass-rate 增益达到 `min_validation_gain`；
72+
3. paired bootstrap 区间下界达到配置值；
73+
4. 不新增 hard failure；
74+
5. key case 不降分；
75+
6. train 提升而 validation 下降时判定 overfit 并拒绝；趋势以 pass-rate 为主，持平时再比较 average score；
76+
7. metric calls、token、耗时和成本均在预算内。
77+
78+
`forbid_new_hard_failures`、`key_cases_no_regression`、`reject_overfitting` 可以显式设为 `false`；报告仍保留检查结果，但 `required=false`，不会参与最终 AND。候选成本从每条 replay trace 的 `usage.cost`/`usage.cost_usd` 汇总并乘以 `num_runs`；任一值缺失时标记为 `unavailable`，成本 Gate fail-closed。仓库自带的离线 trace、judge 和 optimizer 均为本地确定性计算，因此样例成本被明确记录为“已测量的 0”，而不是把未知误写成 0。
79+
80+
## 统计与数据污染防护
81+
82+
- baseline 与 candidate 在同一 case 上配对，使用固定 seed 的 2,000 次 bootstrap，报告 validation pass-rate delta 的 95% 区间。
83+
- 先过安全 Gate，再在合格候选中按 validation 质量、token 与 P95 latency 标记 Pareto 前沿。
84+
- 运行前检查重复 id、train/validation 精确重复、去空白后的重复、相似度 ≥0.92 的近重复，以及 baseline/candidate prompt 中直接出现 validation reference answer；命中即 fail-closed。
85+
- baseline、每个候选和每个 split 的 trace 均保存到 artifact 目录，可由 `AgentEvaluator` 重新播放。
86+
87+
## 输出
88+
89+
```text
90+
runs/<timestamp>/
91+
├── optimization_report.json 机器可读的完整闭环报告
92+
├── optimization_report.md 人类可读决策报告
93+
├── optimizer/ AgentOptimizer 原始 result/round artifacts
94+
└── traces/
95+
├── baseline/{train,validation}.trace.evalset.json
96+
├── ineffective/{train,validation}.trace.evalset.json
97+
├── overfit/{train,validation}.trace.evalset.json
98+
└── robust/{train,validation}.trace.evalset.json
99+
```
100+
101+
JSON 顶层包含 `baseline`、`candidates`、选中 `candidate`、`delta`、`gate`、`failure_attribution`、`optimizer`、`data_quality` 与 `audit`。每个候选保存完整 prompt、SHA-256、seed、来源轮次、逐 case delta、失败证据、metric/judge 调用数、token、P95 latency、耗时与成本。
102+
103+
仓库内的固定示例报告见 `sample_output/`。
104+
105+
## 验证
106+
107+
```bash
108+
pytest -q tests/evaluation/test_eval_optimize_loop_example.py
109+
```
110+
111+
测试覆盖无 Key 全链路、全部候选独立复评、六类失败归因与反误判、score-only 过拟合、paired bootstrap、Pareto、近重复数据拒绝、可配置保护项、成本 fail-closed 和逐候选资源审计。离线完整流程通常在数秒内完成，远低于 3 分钟验收上限。
112+
113+
## 接入真实业务
114+
115+
保留 `EvalOptimizePipeline.run()` 这个外部 interface，将 `offline.py` 的三个 adapter 替换为业务实现：`call_agent` 驱动真实 Agent，reflection/judge 的 `provider_name` 改为实际 provider，trace 物化器读取业务运行日志。`optimizer.json`、`TargetPrompt`、逐候选复评、diff、Gate 和报告模型无需改变。生产环境建议每次使用唯一 output directory；离线 registry 使用进程级固定 replay 配置，不支持在同一 Python 进程中并发启动两个 pipeline。

‎examples/optimization/eval_optimize_loop/__init__.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#
2+
# Tencent is pleased to support the open source community by making trpc-agent-python available.
3+
#
4+
# Copyright (C) 2026 Tencent. All rights reserved.
5+
#
6+
# trpc-agent-python is licensed under the Apache License Version 2.0.
7+
#
8+
"""Evaluation + optimization regression-loop example."""

‎examples/optimization/eval_optimize_loop/agent/__init__.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#
2+
# Tencent is pleased to support the open source community by making trpc-agent-python available.
3+
#
4+
# Copyright (C) 2026 Tencent. All rights reserved.
5+
#
6+
# trpc-agent-python is licensed under the Apache License Version 2.0.
7+
#
8+
"""Prompt assets for the offline support-agent scenario."""

‎examples/optimization/eval_optimize_loop/agent/prompts/candidates/ineffective.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[variant: ineffective]
2+
3+
You are a helpful customer-support agent. Be friendly and concise.

‎examples/optimization/eval_optimize_loop/agent/prompts/candidates/overfit.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[variant: overfit]
2+
3+
You are a customer-support agent tuned to the three training examples. Memorize their answers and prefer them over general rules.

‎examples/optimization/eval_optimize_loop/agent/prompts/candidates/robust.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[variant: robust]
2+
3+
You are a reliable customer-support agent. Select the correct tool, preserve exact identifiers and arguments, ground the answer in retrieved evidence, follow the requested format, and include every required step in order.

‎examples/optimization/eval_optimize_loop/agent/prompts/system.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[variant: baseline]
2+
3+
You are a customer-support agent. Give a short answer from the available evidence.

‎examples/optimization/eval_optimize_loop/data/train.evalset.json‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
{
2+
"eval_set_id": "support_regression_train",
3+
"name": "离线客服回归训练集",
4+
"description": "三个训练 case，分别暴露最终回答、工具选择和参数错误。",
5+
"eval_cases": [
6+
{
7+
"eval_id": "train_final_response_mismatch",
8+
"conversation": [
9+
{
10+
"invocation_id": "expected-1",
11+
"user_content": {
12+
"role": "user",
13+
"parts": [{"text": "请确认售后工单 T-100 的状态。"}]
14+
},
15+
"final_response": {
16+
"role": "model",
17+
"parts": [{"text": "工单 T-100 状态：已完成。"}]
18+
}
19+
}
20+
],
21+
"session_input": {
22+
"app_name": "eval_optimize_loop",
23+
"user_id": "offline",
24+
"state": {
25+
"key_case": false,
26+
"hard_fail": false,
27+
"variant_traces": {
28+
"baseline": {
29+
"final_response": {"role": "model", "parts": [{"text": "工单 T-100 状态：处理中。"}]},
30+
"intermediate_data": {"tool_uses": [], "tool_responses": []},
31+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
32+
"usage": {"input_tokens": 36, "output_tokens": 14, "cost": 0.0, "latency_ms": 4}
33+
},
34+
"ineffective": {
35+
"final_response": {"role": "model", "parts": [{"text": "工单 T-100 状态：处理中。"}]},
36+
"intermediate_data": {"tool_uses": [], "tool_responses": []},
37+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
38+
"usage": {"input_tokens": 38, "output_tokens": 14, "cost": 0.0, "latency_ms": 4}
39+
},
40+
"overfit": {
41+
"final_response": {"role": "model", "parts": [{"text": "工单 T-100 状态：已完成。"}]},
42+
"intermediate_data": {"tool_uses": [], "tool_responses": []},
43+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
44+
"usage": {"input_tokens": 42, "output_tokens": 14, "cost": 0.0, "latency_ms": 5}
45+
},
46+
"robust": {
47+
"final_response": {"role": "model", "parts": [{"text": "工单 T-100 状态：已完成。"}]},
48+
"intermediate_data": {"tool_uses": [], "tool_responses": []},
49+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
50+
"usage": {"input_tokens": 48, "output_tokens": 14, "cost": 0.0, "latency_ms": 5}
51+
}
52+
}
53+
}
54+
}
55+
},
56+
{
57+
"eval_id": "train_tool_call_error",
58+
"conversation": [
59+
{
60+
"invocation_id": "expected-2",
61+
"user_content": {
62+
"role": "user",
63+
"parts": [{"text": "查询订单 O-200 的物流状态。"}]
64+
},
65+
"final_response": {
66+
"role": "model",
67+
"parts": [{"text": "订单 O-200 状态：已发货。"}]
68+
},
69+
"intermediate_data": {
70+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-200"}}],
71+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"order_id": "O-200", "status": "已发货"}}]
72+
}
73+
}
74+
],
75+
"session_input": {
76+
"app_name": "eval_optimize_loop",
77+
"user_id": "offline",
78+
"state": {
79+
"key_case": false,
80+
"hard_fail": true,
81+
"variant_traces": {
82+
"baseline": {
83+
"final_response": {"role": "model", "parts": [{"text": "无法查询订单状态。"}]},
84+
"intermediate_data": {
85+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-200"}}],
86+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"error": "backend unavailable"}}]
87+
},
88+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
89+
"usage": {"input_tokens": 40, "output_tokens": 12, "cost": 0.0, "latency_ms": 7}
90+
},
91+
"ineffective": {
92+
"final_response": {"role": "model", "parts": [{"text": "无法查询订单状态。"}]},
93+
"intermediate_data": {
94+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-200"}}],
95+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"error": "backend unavailable"}}]
96+
},
97+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
98+
"usage": {"input_tokens": 42, "output_tokens": 12, "cost": 0.0, "latency_ms": 7}
99+
},
100+
"overfit": {
101+
"final_response": {"role": "model", "parts": [{"text": "订单 O-200 状态：已发货。"}]},
102+
"intermediate_data": {
103+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-200"}}],
104+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"order_id": "O-200", "status": "已发货"}}]
105+
},
106+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
107+
"usage": {"input_tokens": 46, "output_tokens": 16, "cost": 0.0, "latency_ms": 6}
108+
},
109+
"robust": {
110+
"final_response": {"role": "model", "parts": [{"text": "订单 O-200 状态：已发货。"}]},
111+
"intermediate_data": {
112+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-200"}}],
113+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"order_id": "O-200", "status": "已发货"}}]
114+
},
115+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
116+
"usage": {"input_tokens": 52, "output_tokens": 16, "cost": 0.0, "latency_ms": 6}
117+
}
118+
}
119+
}
120+
}
121+
},
122+
{
123+
"eval_id": "train_parameter_error",
124+
"conversation": [
125+
{
126+
"invocation_id": "expected-3",
127+
"user_content": {
128+
"role": "user",
129+
"parts": [{"text": "查询订单 O-300 的物流状态。"}]
130+
},
131+
"final_response": {
132+
"role": "model",
133+
"parts": [{"text": "订单 O-300 状态：处理中。"}]
134+
},
135+
"intermediate_data": {
136+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-300"}}],
137+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"order_id": "O-300", "status": "处理中"}}]
138+
}
139+
}
140+
],
141+
"session_input": {
142+
"app_name": "eval_optimize_loop",
143+
"user_id": "offline",
144+
"state": {
145+
"key_case": false,
146+
"hard_fail": true,
147+
"variant_traces": {
148+
"baseline": {
149+
"final_response": {"role": "model", "parts": [{"text": "订单 O-399 状态：已取消。"}]},
150+
"intermediate_data": {
151+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-399"}}],
152+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"order_id": "O-399", "status": "已取消"}}]
153+
},
154+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
155+
"usage": {"input_tokens": 40, "output_tokens": 16, "cost": 0.0, "latency_ms": 5}
156+
},
157+
"ineffective": {
158+
"final_response": {"role": "model", "parts": [{"text": "订单 O-399 状态：已取消。"}]},
159+
"intermediate_data": {
160+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-399"}}],
161+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"order_id": "O-399", "status": "已取消"}}]
162+
},
163+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
164+
"usage": {"input_tokens": 42, "output_tokens": 16, "cost": 0.0, "latency_ms": 5}
165+
},
166+
"overfit": {
167+
"final_response": {"role": "model", "parts": [{"text": "订单 O-300 状态：处理中。"}]},
168+
"intermediate_data": {
169+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-300"}}],
170+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"order_id": "O-300", "status": "处理中"}}]
171+
},
172+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
173+
"usage": {"input_tokens": 46, "output_tokens": 16, "cost": 0.0, "latency_ms": 5}
174+
},
175+
"robust": {
176+
"final_response": {"role": "model", "parts": [{"text": "订单 O-300 状态：处理中。"}]},
177+
"intermediate_data": {
178+
"tool_uses": [{"id": "call-1", "name": "lookup_order", "args": {"order_id": "O-300"}}],
179+
"tool_responses": [{"id": "call-1", "name": "lookup_order", "response": {"order_id": "O-300", "status": "处理中"}}]
180+
},
181+
"signals": {"format_pass": true, "knowledge_recall_pass": true, "llm_rubric_pass": true},
182+
"usage": {"input_tokens": 52, "output_tokens": 16, "cost": 0.0, "latency_ms": 5}
183+
}
184+
}
185+
}
186+
}
187+
}
188+
]
189+
}

0 commit comments

Comments
 (0)