feat(examples): add eval-optimize-loop closed-loop optimization pipeline#104
feat(examples): add eval-optimize-loop closed-loop optimization pipeline#104woshidage77 wants to merge 12 commits into
Conversation
Implement 6-phase automatic optimization closed loop for tRPC-Agent: evaluation -> failure attribution -> prompt optimization -> regression verification -> acceptance gate -> audit trail. - Phase 1: Baseline evaluation with fake/real dual-mode support - Phase 2: 6-category failure attribution with 4-layer rule chain - Phase 3: attribution-driven prompt optimization (FakeOptimizer) - Phase 4: Candidate validation with delta matrix - Phase 5: Configurable acceptance gate (5 rules + overfit detection) - Phase 6: Audit trail with JSON + Markdown dual-format reports Includes 99 unit tests covering all phases and full pipeline integration. Fake mode enables complete pipeline execution without API keys. Related: Issue trpc-group#6 (犀牛鸟 2026)
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #104 +/- ##
==========================================
Coverage ? 87.90237%
==========================================
Files ? 479
Lines ? 44984
Branches ? 0
==========================================
Hits ? 39542
Misses ? 5442
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I have read the CLA Document and I hereby sign the CLA |
…ator path - Replace NotImplementedError in optimizer._run_real with AgentOptimizer.optimize() call - Add validator._run_real using AgentEvaluator.evaluate_eval_set() with fallback - Add call_agent.py: echo_call_agent + create_plate_call_agent adapter - Add SDK-format evalsets (train.sdk / val.sdk) and optimizer.sdk.json - Add DESIGN.md (400-char design note) - Harden auditor: threading.Lock() + atomic writes (_atomic_write_text) + .tmp cleanup - Add --mode real-agent CLI option to run_pipeline.py - Keep fake mode intact as fast smoke-test fallback (99 tests pass)
AI Code Review已确认。现在让我写下审查结论。 发现的问题🚨 Critical
|
…, encoding, hashlib Critical: - Wrap Phase 1-6 in try/finally to prevent lock directory leak on exception Warnings: - Plumb --mode arg to BaselineRunner/OptimizationRunner/ValidationRunner - Fix overfit detection: derive candidate train scores separately - Replace builtin hash() with hashlib.sha256 for deterministic cross-process id - Replace image_id positional lookup with path-based case mapping - Fix garbled Chinese in BASE_PROMPTS, CATEGORY_OPTIMIZATION_HINTS, and prompt files Suggestion: - Read critical case ids dynamically from val.evalset.json critical field
AI Code Review发现的问题🚨 Critical
|
Critical: - Rewrite BASE_PROMPTS with readable English content (was all ? chars) - Rewrite CATEGORY_OPTIMIZATION_HINTS with readable English content - Fix SDK config JSONs (train.sdk, val.sdk, optimizer.sdk) with clean fields - Fix test assertions to check real section headers (Workflow, Preprocessing Guide, etc) Warning: - Remove unimplemented --mode trace from CLI choices 99 tests pass
AI Code Review我已经获取了所需的行号。注意 发现的问题🚨 Critical
|
…implify overfit detection CLI gate prevents cryptic NotImplementedError when --mode real or --mode real-agent is used. Fake mode remains the only active path with all 99 tests passing. Changes (AI review round 3): - Add explicit CLI rejection for real/real-agent modes - Remove dead real-agent OptimizationRunner branch - Simplify BaselineRunner/ValidationRunner to always use fake - Unify overfit detection to simulated approach (real branch was broken: reused BaselineRunner without optimized prompt)
AI Code Review基于对 发现的问题🚨 Critical
|
…arning items Critical: - Replace mkdir lock with PID-file lock (survives SIGKILL) - Expand fake-mode gate comments (clarify simulated values) - Add C3 comment on baseline _run_real_split placeholder Warnings: - Plumb run_mode through BaselineRunner/OptimizationRunner/ValidationRunner - Differentiate CANDIDATE_PREDICTIONS per failure category - Derive dominant_condition from real case.conditions (not hardcoded map) - Add FutureWarning when constructing real-mode runners - Add logging.exception in call_agent fallback path - Clarify gate majority strategy (strict majority, ties=reject) - Warn on _read_critical_case_ids fallback - Split long lines in auditor build_trail - Comment auditor latency_ms as fake-mode placeholder
AI Code Review现在让我确认一下测试中关于 对于 现在让我检查 现在让我检查一个真正的问题： 让我验证最具体的 bug。让我检查 现在让我检查 让我检查一个具体的关键 bug： 让我检查 JSON 报告写入器中 在 现在让我检查关键问题： 现在让我检查更令人担忧的安全项： 让我检查 现在让我看看我发现的最具体的正确性 bug。让我检查 让我检查 过拟合检测的真实 bug：在 让我检查 候选者预测中包含 现在让我检查最具体的 bug： 现在让我确认实际的问题。让我重新审视 让我检查 现在让我评估 让我验证一个真正的问题： 让我检查 现在让我检查最重要的一项 —— 现在让我检查 让我检查我标记的一个具体正确性 bug： 让我检查 现在让我确认一个真正的 bug。重新阅读 让我最后检查一下 让我检查 让我检查最后一个真正的问题 —— 让我验证 在 fake_model.py 中，case_id 提取存在 bug： 现在让我检查真正可操作的安全/正确性问题。让我重新审视 让我完成审查。让我检查一个我可能遗漏的更重要的正确性 bug —— 让我验证 让我验证 让我检查 现在让我确认真正的发现。让我更仔细地检查 现在让我看看是否有任何关键问题。让我检查 基于我的分析，让我编写审查报告。关键问题很少；主要是一些警告和建议。让我确定： 警告发现：
让我验证 #6 是否真的有影响 —— 审计通过 让我检查 让我验证 让我确定并撰写审查报告。让我再次检查 #5 —— 回退到 让我验证一个更具体的 bug： 现在让我撰写审查报告。 发现的问题🚨 Critical
|
…arning + 1 suggestion Critical: - _read_critical_case_ids: return [] on error instead of guessing [val_001] Warnings: - PID lock: only remove lock if our PID owns it (TOCTOU mitigation) - _make_candidate_id: remove time.time(), use deterministic hash+iteration - sys.path: pop before re-raising ImportError in baseline _run_real_split - Gate overfit: note fake-mode simulation in check description - candidate_train_scores: documented as fake placeholder Suggestion: - BaselineCaseResult.to_dict: add trajectory field
AI Code Review发现的问题🚨 Critical
|
…arning + 1 suggestion Critical: - _pid_alive: fix Linux dead-PID false-positive (ctypes.windll AttributeError was caught by blind except Exception, returning True for all dead processes. Now: ProcessLookupError -> dead, PermissionError -> alive, platform-gated Windows ctypes branch, Unix returns os.kill result) Warnings: - --max-iter: plumb args.max_iter to OptimizationRunner config - Lock file: use output_dir instead of hardcoded BASE_DIR/output - test_knowledge_recall_from_trajectory: assert exact category - test_param_error_from_trajectory: assert priority-based classification - baseline image_id hash: retain C3 placeholder comment (no code change) Suggestion: - auditor: rename total_latency_ms -> avg_latency_ms (semantic fix)
AI Code Review现在我已经有足够的信息来撰写审查报告了。让我整合一下我的发现。 发现的问题🚨 Critical
|
…arning + 1 suggestion
Critical:
- PID lock: atomic write via temp file + fsync + os.replace (was non-atomic
open('w') without flush, partial writes on crash)
- real mode PlateEvaluator: ACK via reply, docstring already marks PLACEHOLDER
Warnings:
- --max-iter: change default from 3 to None, use is not None check
- gate all_must_pass: reject when checks list is empty (was all([])=True)
- test_knowledge_recall: assert exact category instead of loose in() + pass
- REGRESSION_PREDICTIONS: differentiate val_002/val_003 from baseline
- baseline image_id: retain existing C3 comment (no code change)
Suggestion:
- Remove unused imports: time from optimizer, BaselineResult/AttributionReport
from auditor, unused AttributionCluster/CATEGORY_META from optimizer
AI Code Review我已经了解了全貌。让我来整理一下审查意见。 发现的问题🚨 Critical
|
…arning Critical: - fake_judge: clamp response_quality to [0.2, 1.0] (was recognition*1.05 could exceed 1.0, causing score overflow in gate and audit reports) Warnings: - PID lock: atomic acquire via os.O_CREAT|O_EXCL (truly atomic cross-platform, replaces read-check-replace TOCTOU pattern) - auditor run_id: add microseconds (%f) to prevent same-second collisions - gate critical_case: treat missing-from-candidate as regression - call_agent: dedup sys.path.insert (only insert if path not present) - baseline _run_real_split: add sys.path cleanup note in docstring - candidate_train_scores simulated + image_id mapping: already documented - Chinese comments: acknowledged, no functional change
AI Code Review我将进行静态审查，无需运行测试。现根据对 diff 的分析，整理审查发现如下。 发现的问题🚨 Critical
|
…arning Critical: - optimizer: cumulative multi-iteration (each iteration now uses previous prompt_after as prompt_before, not the original BASE_PROMPTS every time) - baseline real mode image_id hash: ACK, docstring already marks PLACEHOLDER Warnings: - run_pipeline: remove unused import time and redundant sys import - auditor: add None guard in save() baseline dict comprehension - gate/attribution: extract PASS_THRESHOLD constant (0.6) from fake_judge, share across gate._check_no_new_hard_fail and attribution judge checks - _pid_alive ctypes: already fixed in round 6, no change needed - candidate_train_scores simulated: already documented Suggestion: - validator: warn on unknown failure_category fallback in CANDIDATE_PREDICTIONS
…warning/suggestion Critical: - baseline.py: replace SHA256-hash image_id with sequential enumerate(start=1) + explicit id_to_case reverse mapping for stable case_id lookup - run_pipeline.py: _pid_alive Windows except Exception now returns False instead of True, preventing permanent stale-lock deadlock Warning: - auditor.py: None check changed from if v to if v is not None - tests/*: 5 fixtures converted from manual asyncio.new_event_loop() to @pytest_asyncio.fixture + async def (avoids pytest-asyncio conflicts) Suggestion: - validator.py: CANDIDATE_PREDICTIONS fallback now emits warnings.warn - optimizer.py: _generate_optimization docstring explicitly marks HTML-comment approach as fake-mode placeholder 99 tests pass, pipeline 6 phases run end-to-end.
3 participants
Implement 6-phase automatic optimization closed loop for tRPC-Agent: evaluation -> failure attribution -> prompt optimization -> regression verification -> acceptance gate -> audit trail.
Includes 99 unit tests covering all phases and full pipeline integration. Fake mode enables complete pipeline execution without API keys.
Related: Issue #6 (犀牛鸟 2026)