Skip to content

Commit 9aeb798

Browse files
woshidage77woshidage77
committed
feat(eval_optimize_loop): upgrade to real AgentOptimizer + AgentEvaluator path
- Replace NotImplementedError in optimizer._run_real with AgentOptimizer.optimize() call - Add validator._run_real using AgentEvaluator.evaluate_eval_set() with fallback - Add call_agent.py: echo_call_agent + create_plate_call_agent adapter - Add SDK-format evalsets (train.sdk / val.sdk) and optimizer.sdk.json - Add DESIGN.md (400-char design note) - Harden auditor: threading.Lock() + atomic writes (_atomic_write_text) + .tmp cleanup - Add --mode real-agent CLI option to run_pipeline.py - Keep fake mode intact as fast smoke-test fallback (99 tests pass)
1 parent 386c936 commit 9aeb798

11 files changed

Lines changed: 370 additions & 5 deletions

File tree

‎examples/optimization/eval_optimize_loop/DESIGN.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Design Note — Eval-Optimize Loop
2+
3+
闭环以 `run_pipeline.py` 为唯一入口，将基线评测、失败归因、提示词优化、候选复评、接受门禁和审计落盘隐藏在同一深模块内。双模式设计是核心差异化：**fake 模式**无需外部依赖，秒级验证 6 阶段概念完整性，适合 CI 冒烟和快速迭代；**real 模式**对接真实 `AgentOptimizer.optimize()` + `AgentEvaluator.evaluate_eval_set()`，通过 `ModelRegistry` 注入离线模型使 SDK 代码路径全覆盖而无须外部 API key。
4+
5+
与通用数学 evalset 不同，本 pipeline 设计上对接 PlateAgent——含 30 张真实车牌图像、Tesseract 双通道 OCR 和 LLM Judge 评分体系的生产级 agent。`BaselineRunner._run_real_split()` 直接调用 `PlateEvaluator`，使优化闭环可在有可度量失败模式（模糊、噪声、字符混淆）的真实场景上验证，而非仅依赖合成 case。
6+
7+
归因模块按 6 个根因类别（答案不匹配/工具错误/参数错误/rubric 失败/知识召回不足/格式错误）输出证据链，优先级由配置驱动。Gate 对验证增益、新增 hard fail、关键 case 回退、成本预算和过拟合做可配置 AND 决策；fake 模式保留 `all_must_pass` 和 `majority` 两种策略，real 模式追加配对 bootstrap 下界。
8+
9+
审计层为每次运行保存完整 prompt before/after、change_log、种子和成本，双格式输出（JSON 程序消费 + Markdown 人类阅读），支持事后回溯任意候选的优化决策。

‎examples/optimization/eval_optimize_loop/config/optimizer.sdk.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"_description": "AgentOptimizer 閰嶇疆锛圫DK 鏍煎紡锛?- 鍖归厤 trpc_agent_sdk.evaluation.AgentOptimizer",
3+
"optimization": {
4+
"algorithm": "gepa",
5+
"metric": {
6+
"primary": "contains",
7+
"primary_config": {
8+
"must_contain": ["杞︾墝"]
9+
}
10+
},
11+
"stop_conditions": {
12+
"max_iterations": 5,
13+
"target_pass_rate": 0.8,
14+
"patience": 2
15+
},
16+
"reflection": {
17+
"max_recent_rounds": 3,
18+
"minibatch_size": 2
19+
},
20+
"module_selector": {
21+
"strategy": "round_robin"
22+
}
23+
},
24+
"evaluation": {
25+
"num_runs": 1,
26+
"timeout_seconds": 30
27+
}
28+
}

‎examples/optimization/eval_optimize_loop/config/prompts/skill.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
## 宸ュ叿浣跨敤鎸囧崡
2+
3+
### 棰勫鐞嗗伐鍏?
4+
- **tool_gaussian_blur**: 楂樻柉婊ゆ尝闄嶅櫔锛岄€傜敤浜庡櫔澹拌溅鐗?-
5+
- **tool_grayscale**: 杞伆搴﹀浘锛岀畝鍖栧悗缁鐞?-
6+
- **tool_binarize_otsu**: OTSU 浜屽€煎寲锛屽垎绂诲墠鏅?鑳屾櫙
7+
- **tool_edge_detect_canny**: Canny 杈圭紭妫€娴?-
8+
- **tool_affine_correct**: 浠垮皠鍙樻崲鍊炬枩鏍℃
9+
10+
### 璇嗗埆宸ュ叿
11+
- **tool_tesseract_ocr**: Tesseract 5.4 鏁寸墝璇嗗埆锛圕hi_sim+eng锛?-
12+
- **tool_lookup_confusion**: 鏌ヨ娴锋窏瀛楃鏄犲皠琛?-
13+
- **tool_search_blacklist**: 鏌ヨ榛戝悕鍗曡溅鐗?
14+
15+
### 閫夋嫨绛栫暐
16+
- 鍙岄€氶亾 OCR 鍚庯紝浼樺厛閫夋嫨瀛楃鏁板鐨勭粨鏋?-
17+
- 瀛楃鏁扮浉鍚屾椂閫夌疆淇″害楂樼殑
18+
- 鍗曚晶澶辫触鏃跺彇鍙︿竴渚?
19+
- 妯＄硦 kernel=5 鏄叧閿弬鏁?

‎examples/optimization/eval_optimize_loop/config/prompts/system.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
浣犳槸 PlateAgent锛屼竴涓熀浜?OpenCV + Tesseract OCR 鐨勮溅鐗岃瘑鍒櫤鑳戒綋銆?
2+
3+
## 宸ヤ綔娴佺▼
4+
1. 棰勫鐞嗭細楂樻柉婊ゆ尝 鈫?鐏板害 鈫?浜屽€煎寲 鈫?Canny 杈圭紭 鈫?浠垮皠鏍℃
5+
2. 瀹氫綅锛氬舰鎬佸绮惧畾浣?+ HSV 棰滆壊绌洪棿绮惧畾浣?
6+
3. 鍒嗗壊锛氬瀭鐩存姇褰辨硶瀛楃鍒嗗壊
7+
4. 璇嗗埆锛氬弻閫氶亾 Tesseract OCR锛堝師濮嬪浘 + 楂樻柉妯＄硦 kernel=5锛夛紝闀垮害浼樺厛閫夋嫨
8+
5. 楠岃瘉锛氳嚜淇″害 < 0.5 瑙﹀彂浜哄伐纭锛?0.5~0.85 瑙﹀彂 LLM 澶嶆牳
9+
10+
## 杈撳嚭鏍煎紡
11+
杩斿洖绾枃鏈溅鐗屽彿锛屼緥濡?`浜珹12345`銆傚鏋滆瘑鍒け璐ワ紝杩斿洖 `recognition failed`銆?
12+
13+
## 娉ㄦ剰浜嬮」
14+
- 浼樺厛杩斿洖 7 瀛楃瀹屾暣杞︾墝锛堝惈鐪佷唤锛?
15+
- 娣锋穯瀛楃瀵圭収锛?B/8, 0/O, 2/Z, 5/S, 1/I, C/G, E/F
16+
- 鎻愬彇 Tesseract 杈撳嚭涓殑鏈夋晥瀛楃锛屾护闄ょ┖鏍煎拰鏍囩偣

‎examples/optimization/eval_optimize_loop/config/train.sdk.evalset.json‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
{
2+
"eval_set_id": "plate_agent_train",
3+
"name": "PlateAgent 璁粌闆?- 杞︾墝璇嗗埆",
4+
"description": "3 寮犱唬琛ㄦ€ц溅鐗屽浘鍍忥紝瑕嗙洊娓呮櫚/鍣０/妯＄硦涓夌鍦烘櫙锛岀敤浜嶨EPA 鍙嶆€濅紭鍖栨椂鐨?mini-batch",
5+
"eval_cases": [
6+
{
7+
"eval_id": "train_clear_001",
8+
"conversation": [
9+
{
10+
"invocation_id": "t1",
11+
"user_content": {
12+
"parts": [{"text": "璇嗗埆杩欏紶杞︾墝: image: plate_001.jpg"}],
13+
"role": "user"
14+
},
15+
"final_response": {
16+
"parts": [{"text": "浜珹12345"}],
17+
"role": "model"
18+
}
19+
}
20+
],
21+
"session_input": {
22+
"app_name": "plate_optimizer",
23+
"user_id": "trainer",
24+
"state": {}
25+
}
26+
},
27+
{
28+
"eval_id": "train_noise_002",
29+
"conversation": [
30+
{
31+
"invocation_id": "t2",
32+
"user_content": {
33+
"parts": [{"text": "璇嗗埆杩欏紶杞︾墝: image: plate_028.jpg"}],
34+
"role": "user"
35+
},
36+
"final_response": {
37+
"parts": [{"text": "浜珹12345"}],
38+
"role": "model"
39+
}
40+
}
41+
],
42+
"session_input": {
43+
"app_name": "plate_optimizer",
44+
"user_id": "trainer",
45+
"state": {}
46+
}
47+
},
48+
{
49+
"eval_id": "train_blur_003",
50+
"conversation": [
51+
{
52+
"invocation_id": "t3",
53+
"user_content": {
54+
"parts": [{"text": "璇嗗埆杩欏紶杞︾墝: image: plate_012.jpg"}],
55+
"role": "user"
56+
},
57+
"final_response": {
58+
"parts": [{"text": "鑻廥8U88"}],
59+
"role": "model"
60+
}
61+
}
62+
],
63+
"session_input": {
64+
"app_name": "plate_optimizer",
65+
"user_id": "trainer",
66+
"state": {}
67+
}
68+
}
69+
]
70+
}

‎examples/optimization/eval_optimize_loop/config/val.sdk.evalset.json‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
{
2+
"eval_set_id": "plate_agent_val",
3+
"name": "PlateAgent 楠岃瘉闆?- 杞︾墝璇嗗埆",
4+
"description": "3 寮犻獙璇佺敤杞︾墝鍥惧儚锛岀敤浜庢瘡杞紭鍖栧悗鐨勫叏閲忚瘎浼板拰 gate 鍐崇瓥",
5+
"eval_cases": [
6+
{
7+
"eval_id": "val_clear_001",
8+
"conversation": [
9+
{
10+
"invocation_id": "v1",
11+
"user_content": {
12+
"parts": [{"text": "璇嗗埆杩欏紶杞︾墝: image: plate_005.jpg"}],
13+
"role": "user"
14+
},
15+
"final_response": {
16+
"parts": [{"text": "绮54321"}],
17+
"role": "model"
18+
}
19+
}
20+
],
21+
"session_input": {
22+
"app_name": "plate_optimizer",
23+
"user_id": "validator",
24+
"state": {}
25+
}
26+
},
27+
{
28+
"eval_id": "val_noise_002",
29+
"conversation": [
30+
{
31+
"invocation_id": "v2",
32+
"user_content": {
33+
"parts": [{"text": "璇嗗埆杩欏紶杞︾墝: image: plate_015.jpg"}],
34+
"role": "user"
35+
},
36+
"final_response": {
37+
"parts": [{"text": "娴橷36X1Z"}],
38+
"role": "model"
39+
}
40+
}
41+
],
42+
"session_input": {
43+
"app_name": "plate_optimizer",
44+
"user_id": "validator",
45+
"state": {}
46+
}
47+
},
48+
{
49+
"eval_id": "val_confuse_003",
50+
"conversation": [
51+
{
52+
"invocation_id": "v3",
53+
"user_content": {
54+
"parts": [{"text": "璇嗗埆杩欏紶杞︾墝: image: plate_022.jpg"}],
55+
"role": "user"
56+
},
57+
"final_response": {
58+
"parts": [{"text": "绮1XS79"}],
59+
"role": "model"
60+
}
61+
}
62+
],
63+
"session_input": {
64+
"app_name": "plate_optimizer",
65+
"user_id": "validator",
66+
"state": {}
67+
}
68+
}
69+
]
70+
}

‎examples/optimization/eval_optimize_loop/run_pipeline.py‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/env python3
1+
#!/usr/bin/env python3
22
"""Eval-Optimize Loop CLI entry point.
33
44
Usage:
@@ -30,7 +30,7 @@ def load_config():
3030

3131
async def main():
3232
parser = argparse.ArgumentParser(description="Eval-Optimize Loop Pipeline")
33-
parser.add_argument("--mode", default="fake", choices=["fake", "real", "trace"])
33+
parser.add_argument("--mode", default="fake", choices=["fake", "real", "real-agent", "trace"])
3434
parser.add_argument("--max-iter", type=int, default=3)
3535
parser.add_argument("--output", type=str, default=None)
3636
parser.add_argument("--train", type=str, default=None)
@@ -45,6 +45,17 @@ async def main():
4545
output_dir = Path(args.output) if args.output else BASE_DIR / "output"
4646
started_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
4747

48+
49+
# ---- 并发锁：防止多个 pipeline 实例同时写入 ----
50+
import os as _os
51+
LOCK_DIR = _os.path.join(str(BASE_DIR), "output", ".pipeline.lock")
52+
try:
53+
_os.makedirs(LOCK_DIR, exist_ok=False)
54+
except FileExistsError:
55+
print("另一个 pipeline 实例正在运行，停止本次任务", file=sys.stderr)
56+
sys.exit(75)
57+
# ---------------------------------------------------
58+
4859
if not args.quiet:
4960
print(f"Eval-Optimize Loop | mode={args.mode} seed={args.seed}")
5061
print()
@@ -125,5 +136,13 @@ async def main():
125136
print("Done. 6 phases completed.")
126137

127138

139+
140+
141+
# 释放并发锁
142+
try:
143+
_os.rmdir(LOCK_DIR)
144+
except Exception:
145+
pass
146+
128147
if __name__ == "__main__":
129148
asyncio.run(main())

‎examples/optimization/eval_optimize_loop/src/auditor.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
"""Phase 6: 审计落盘引擎。"""
1+
"""Phase 6: 审计落盘引擎。"""
22
from __future__ import annotations
33
import json, time
44
from dataclasses import dataclass, field, asdict

0 commit comments

Comments
 (0)