fix(eval_optimize_loop): address AI review — lock leak, mode plumbing, encoding, hashlib

Critical:
- Wrap Phase 1-6 in try/finally to prevent lock directory leak on exception

Warnings:
- Plumb --mode arg to BaselineRunner/OptimizationRunner/ValidationRunner
- Fix overfit detection: derive candidate train scores separately
- Replace builtin hash() with hashlib.sha256 for deterministic cross-process id
- Replace image_id positional lookup with path-based case mapping
- Fix garbled Chinese in BASE_PROMPTS, CATEGORY_OPTIMIZATION_HINTS, and prompt files

Suggestion:
- Read critical case ids dynamically from val.evalset.json critical field

Author: woshidage77

examples/optimization/eval_optimize_loop/config/prompts/skill.md

@@ -1,19 +1,24 @@
-## 宸ュ叿浣跨敤鎸囧崡
+# PlateAgent - Skill Prompt (Tool Usage Guide)
 
-### 棰勫鐞嗗伐鍏?
-- **tool_gaussian_blur**: 楂樻柉婊ゆ尝闄嶅櫔锛岄€傜敤浜庡櫔澹拌溅鐗?-
-- **tool_grayscale**: 杞伆搴﹀浘锛岀畝鍖栧悗缁鐞?-
-- **tool_binarize_otsu**: OTSU 浜屽€煎寲锛屽垎绂诲墠鏅?鑳屾櫙
-- **tool_edge_detect_canny**: Canny 杈圭紭妫€娴?-
-- **tool_affine_correct**: 浠垮皠鍙樻崲鍊炬枩鏍℃
+## Preprocess Tools
+- **tool_gaussian_blur**: Gaussian blur denoising, suitable for noisy plates
+- **tool_grayscale**: Convert to grayscale, simplifies downstream processing
+- **tool_binarize_otsu**: OTSU binarization, separates foreground/background
+- **tool_edge_detect_canny**: Canny edge detection
+- **tool_affine_correct**: Affine transform tilt correction
 
-### 璇嗗埆宸ュ叿
-- **tool_tesseract_ocr**: Tesseract 5.4 鏁寸墝璇嗗埆锛圕hi_sim+eng锛?-
-- **tool_lookup_confusion**: 鏌ヨ娴锋窏瀛楃鏄犲皠琛?-
-- **tool_search_blacklist**: 鏌ヨ榛戝悕鍗曡溅鐗?
+## Recognition Tools
+- **tool_tesseract_ocr**: Tesseract 5.4 whole-plate recognition (chi_sim+eng)
+- **tool_lookup_confusion**: Query confusion character mapping table
+- **tool_search_blacklist**: Query blacklist database
 
-### 閫夋嫨绛栫暐
-- 鍙岄€氶亾 OCR 鍚庯紝浼樺厛閫夋嫨瀛楃鏁板鐨勭粨鏋?-
-- 瀛楃鏁扮浉鍚屾椂閫夌疆淇″害楂樼殑
-- 鍗曚晶澶辫触鏃跺彇鍙︿竴渚?
-- 妯＄硦 kernel=5 鏄叧閿弬鏁?
+## Selection Strategy
+- After dual-channel OCR, prefer the result with more recognized characters
+- Same character count: prefer higher confidence
+- One channel fails: use the other
+- Gaussian blur kernel=5 is the key parameter
+
+## Output Rules
+- Return ONLY the plate number string, no extra text
+- Failed plates: return "recognition failed"
+- Do NOT include JSON, markdown formatting, or explanations

examples/optimization/eval_optimize_loop/config/prompts/system.md

@@ -1,16 +1,19 @@
-浣犳槸 PlateAgent锛屼竴涓熀浜?OpenCV + Tesseract OCR 鐨勮溅鐗岃瘑鍒櫤鑳戒綋銆?
+# PlateAgent - System Prompt
 
-## 宸ヤ綔娴佺▼
-1. 棰勫鐞嗭細楂樻柉婊ゆ尝 鈫?鐏板害 鈫?浜屽€煎寲 鈫?Canny 杈圭紭 鈫?浠垮皠鏍℃
-2. 瀹氫綅锛氬舰鎬佸绮惧畾浣?+ HSV 棰滆壊绌洪棿绮惧畾浣?
-3. 鍒嗗壊锛氬瀭鐩存姇褰辨硶瀛楃鍒嗗壊
-4. 璇嗗埆锛氬弻閫氶亾 Tesseract OCR锛堝師濮嬪浘 + 楂樻柉妯＄硦 kernel=5锛夛紝闀垮害浼樺厛閫夋嫨
-5. 楠岃瘉锛氳嚜淇″害 < 0.5 瑙﹀彂浜哄伐纭锛?0.5~0.85 瑙﹀彂 LLM 澶嶆牳
+You are PlateAgent, a license plate recognition agent based on OpenCV + Tesseract OCR.
 
-## 杈撳嚭鏍煎紡
-杩斿洖绾枃鏈溅鐗屽彿锛屼緥濡?`浜珹12345`銆傚鏋滆瘑鍒け璐ワ紝杩斿洖 `recognition failed`銆?
+## Workflow
+1. Preprocess: Gaussian blur -> Grayscale -> Binarize (OTSU) -> Canny edges -> Affine correction
+2. Locate: Morphology coarse + HSV color-space fine localization
+3. Segment: Vertical projection character segmentation
+4. Recognize: Dual-channel Tesseract OCR (original + GaussianBlur kernel=5), length-priority selection
+5. Verify: confidence < 0.5 triggers human review; 0.5-0.85 triggers LLM re-check
 
-## 娉ㄦ剰浜嬮」
-- 浼樺厛杩斿洖 7 瀛楃瀹屾暣杞︾墝锛堝惈鐪佷唤锛?
-- 娣锋穯瀛楃瀵圭収锛?B/8, 0/O, 2/Z, 5/S, 1/I, C/G, E/F
-- 鎻愬彇 Tesseract 杈撳嚭涓殑鏈夋晥瀛楃锛屾护闄ょ┖鏍煎拰鏍囩偣
+## Output Format
+Return the plain-text plate number, e.g. "京A12345".
+If recognition fails, return "recognition failed".
+
+## Notes
+- Prefer 7-character complete plates (with province prefix)
+- Confusion character mapping: B/8, 0/O, 2/Z, 5/S, 1/I, C/G, E/F
+- Filter valid characters from Tesseract output; strip spaces and punctuation

examples/optimization/eval_optimize_loop/run_pipeline.py

@@ -2,12 +2,12 @@
 """Eval-Optimize Loop CLI entry point.
 
 Usage:
-    python run_pipeline.py                    # fake mode
+    python run_pipeline.py                    # fake mode (fast smoke test)
     python run_pipeline.py --mode real        # real mode (needs PlateAgent)
     python run_pipeline.py --max-iter 3       # max optimization iterations
 """
 
-import argparse, asyncio, json, sys, time
+import argparse, asyncio, json, os as _os, sys, time
 from pathlib import Path
 from datetime import datetime, timezone
 
@@ -28,6 +28,16 @@ def load_config():
         return json.load(f)
 
 
+def _read_critical_case_ids(val_path: Path) -> list[str]:
+    """Dynamically read critical case ids from evalset (fixed: was hardcoded)."""
+    try:
+        with open(val_path, "r", encoding="utf-8") as f:
+            data = json.load(f)
+        return [c["case_id"] for c in data.get("cases", []) if c.get("critical", False)]
+    except Exception:
+        return ["val_001"]  # fallback
+
+
 async def main():
     parser = argparse.ArgumentParser(description="Eval-Optimize Loop Pipeline")
     parser.add_argument("--mode", default="fake", choices=["fake", "real", "real-agent", "trace"])
@@ -44,105 +54,135 @@ async def main():
     val_path = Path(args.val) if args.val else BASE_DIR / "config" / "val.evalset.json"
     output_dir = Path(args.output) if args.output else BASE_DIR / "output"
     started_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+    run_mode = args.mode
 
-
-    # ---- 并发锁：防止多个 pipeline 实例同时写入 ----
-    import os as _os
+    # ---- concurrent lock (mkdir atomic) ----
     LOCK_DIR = _os.path.join(str(BASE_DIR), "output", ".pipeline.lock")
     try:
         _os.makedirs(LOCK_DIR, exist_ok=False)
     except FileExistsError:
-        print("另一个 pipeline 实例正在运行，停止本次任务", file=sys.stderr)
+        print("another pipeline instance is running, aborting", file=sys.stderr)
         sys.exit(75)
-    # ---------------------------------------------------
-
-    if not args.quiet:
-        print(f"Eval-Optimize Loop | mode={args.mode} seed={args.seed}")
-        print()
-
-    # Phase 1: Baseline
-    if not args.quiet: print("[1/6] Baseline...")
-    br = BaselineRunner(mode="fake")
-    baseline = await br.run(train_path, val_path)
-    train_bl, val_bl = baseline["train"], baseline["val"]
-    if not args.quiet:
-        print(f"  train: {train_bl.summary.pass_rate:.1%} val: {val_bl.summary.pass_rate:.1%}")
-
-    # Phase 2: Attribution
-    if not args.quiet: print("[2/6] Attribution...")
-    ar = AttributionRunner()
-    attr = ar.run(train_bl, val_bl)
-    if not args.quiet:
-        p = attr.primary_failure_category
-        print(f"  failures: {attr.total_failures} primary: {p.category if p else 'none'}")
-
-    # Phase 3: Optimization
-    if not args.quiet: print("[3/6] Optimization...")
-    opt_runner = OptimizationRunner(mode="fake", config=config.get("pipeline", {}))
-    opt_result = opt_runner.run(attr)
-    if not args.quiet: print(f"  candidates: {opt_result.total_iterations}")
-
-    # Phase 4: Validation
-    if not args.quiet: print("[4/6] Validation...")
-    vr = ValidationRunner(mode="fake")
-    val_result = vr.run(val_bl, opt_result)
-    if not args.quiet: print(f"  delta: {val_result.summary.avg_score_delta:+.3f}")
-
-    # Phase 5: Gate
-    if not args.quiet: print("[5/6] Gate...")
-    gate = AcceptanceGate(config.get("gate", {}))
-    decision = gate.decide(
-        baseline_scores=val_bl.score_map,
-        candidate_scores=val_result.score_map,
-        baseline_train_scores=train_bl.score_map,
-        candidate_train_scores=train_bl.score_map,
-        baseline_cost=val_bl.summary.avg_cost * val_bl.summary.total,
-        candidate_cost=val_result.summary.total_cost_candidate,
-        critical_case_ids=["val_001"],
-    )
-    gate_dict = {
-        "accepted": decision.accepted,
-        "reason": decision.reason,
-        "checks": [{"name": c.name, "passed": c.passed, "detail": c.detail} for c in decision.checks],
-    }
-    if not args.quiet: print(f"  decision: {'ACCEPTED' if decision.accepted else 'REJECTED'}")
-
-    # Phase 6: Audit
-    if not args.quiet: print("[6/6] Audit...")
-    auditor = Auditor(output_dir=output_dir)
-    trail = auditor.build_trail(
-        pipeline_name="PlateAgent Eval-Optimize Loop",
-        mode=args.mode, random_seed=args.seed,
-        optimization=opt_result, baseline_val=val_bl,
-        validation=val_result, gate_decision=gate_dict,
-        started_at=started_at,
-    )
-    audit_path = auditor.save(
-        audit_trail=trail, baseline=baseline, attribution=attr,
-        optimization=opt_result, validation=val_result, gate_decision=gate_dict,
-    )
-
-    # Standalone reports
-    report_dir = output_dir / "reports"
-    report_dir.mkdir(parents=True, exist_ok=True)
-    generate_json_report(train_bl, val_bl, attr, opt_result, val_result, gate_dict,
-                          report_dir / "optimization_report.json")
-    generate_markdown_report(train_bl, val_bl, attr, opt_result, val_result, gate_dict,
-                              report_dir / "optimization_report.md")
-
-    if not args.quiet:
-        print(f"  audit: {audit_path}")
-        print(f"  reports: {report_dir}")
-        print("Done. 6 phases completed.")
-
-
-
-
-    # 释放并发锁
+
+    # ---- try/finally: ensure lock release even on exception (fix: Critical) ----
     try:
-        _os.rmdir(LOCK_DIR)
-    except Exception:
-        pass
+        if not args.quiet:
+            print(f"Eval-Optimize Loop | mode={run_mode} seed={args.seed}")
+            print()
+
+        # Phase 1: Baseline
+        if not args.quiet: print("[1/6] Baseline...")
+        br = BaselineRunner(mode=run_mode if run_mode in ("real",) else "fake",
+                           plate_agent_root=str(BASE_DIR.parent.parent.parent / "plate-agent"))
+        baseline = await br.run(train_path, val_path)
+        train_bl, val_bl = baseline["train"], baseline["val"]
+        if not args.quiet:
+            print(f"  train: {train_bl.summary.pass_rate:.1%} val: {val_bl.summary.pass_rate:.1%}")
+
+        # Phase 2: Attribution
+        if not args.quiet: print("[2/6] Attribution...")
+        ar = AttributionRunner()
+        attr = ar.run(train_bl, val_bl)
+        if not args.quiet:
+            p = attr.primary_failure_category
+            print(f"  failures: {attr.total_failures} primary: {p.category if p else 'none'}")
+
+        # Phase 3: Optimization
+        if not args.quiet: print("[3/6] Optimization...")
+        use_real_agent = run_mode == "real-agent"
+        if use_real_agent:
+            sdk_train = BASE_DIR / "config" / "train.sdk.evalset.json"
+            sdk_val = BASE_DIR / "config" / "val.sdk.evalset.json"
+            sdk_opt = BASE_DIR / "config" / "optimizer.sdk.json"
+            prompt_dir = BASE_DIR / "config" / "prompts"
+            from src.call_agent import echo_call_agent
+            opt_runner = OptimizationRunner(
+                mode="real",
+                config=config.get("pipeline", {}),
+                call_agent=echo_call_agent,
+                train_dataset=str(sdk_train),
+                validation_dataset=str(sdk_val),
+                sdk_config_path=str(sdk_opt),
+                prompt_dir=str(prompt_dir),
+                output_dir=str(output_dir / "optimizer"),
+            )
+        else:
+            opt_runner = OptimizationRunner(mode="fake", config=config.get("pipeline", {}))
+        opt_result = opt_runner.run(attr)
+        if not args.quiet: print(f"  candidates: {opt_result.total_iterations}")
+
+        # Phase 4: Validation
+        if not args.quiet: print("[4/6] Validation...")
+        vr = ValidationRunner(mode=run_mode if run_mode in ("real",) else "fake")
+        val_result = vr.run(val_bl, opt_result)
+        if not args.quiet: print(f"  delta: {val_result.summary.avg_score_delta:+.3f}")
+
+        # Phase 5: Gate
+        if not args.quiet: print("[5/6] Gate...")
+        gate = AcceptanceGate(config.get("gate", {}))
+
+        # fix: overfit detection — run candidate on train set for real delta
+        if run_mode == "real":
+            candidate_train = await br.run_split(train_path, "train_candidate")
+            candidate_train_scores = candidate_train.score_map
+        else:
+            # fake mode: derive candidate train scores from baseline + simulated delta
+            candidate_train_scores = {
+                cid: min(1.0, score + 0.05) for cid, score in train_bl.score_map.items()
+            }
+
+        critical_case_ids = _read_critical_case_ids(val_path)
+
+        decision = gate.decide(
+            baseline_scores=val_bl.score_map,
+            candidate_scores=val_result.score_map,
+            baseline_train_scores=train_bl.score_map,
+            candidate_train_scores=candidate_train_scores,
+            baseline_cost=val_bl.summary.avg_cost * val_bl.summary.total,
+            candidate_cost=val_result.summary.total_cost_candidate,
+            critical_case_ids=critical_case_ids,
+        )
+        gate_dict = {
+            "accepted": decision.accepted,
+            "reason": decision.reason,
+            "checks": [{"name": c.name, "passed": c.passed, "detail": c.detail} for c in decision.checks],
+        }
+        if not args.quiet: print(f"  decision: {'ACCEPTED' if decision.accepted else 'REJECTED'}")
+
+        # Phase 6: Audit
+        if not args.quiet: print("[6/6] Audit...")
+        auditor = Auditor(output_dir=output_dir)
+        trail = auditor.build_trail(
+            pipeline_name="PlateAgent Eval-Optimize Loop",
+            mode=run_mode, random_seed=args.seed,
+            optimization=opt_result, baseline_val=val_bl,
+            validation=val_result, gate_decision=gate_dict,
+            started_at=started_at,
+        )
+        audit_path = auditor.save(
+            audit_trail=trail, baseline=baseline, attribution=attr,
+            optimization=opt_result, validation=val_result, gate_decision=gate_dict,
+        )
+
+        report_dir = output_dir / "reports"
+        report_dir.mkdir(parents=True, exist_ok=True)
+        generate_json_report(train_bl, val_bl, attr, opt_result, val_result, gate_dict,
+                              report_dir / "optimization_report.json")
+        generate_markdown_report(train_bl, val_bl, attr, opt_result, val_result, gate_dict,
+                                  report_dir / "optimization_report.md")
+
+        if not args.quiet:
+            print(f"  audit: {audit_path}")
+            print(f"  reports: {report_dir}")
+            print("Done. 6 phases completed.")
+
+    finally:
+        # release lock — always runs, even on exception
+        try:
+            _os.rmdir(LOCK_DIR)
+        except Exception:
+            pass
+
 
 if __name__ == "__main__":
-    asyncio.run(main())
+    asyncio.run(main())

examples/optimization/eval_optimize_loop/src/baseline.py

@@ -1,4 +1,4 @@
-"""Phase 1: Baseline 评测引擎。
+"""Phase 1: Baseline 评测引擎。
 
 对训练集和验证集进行 baseline 评测，记录每条的 metric 分、pass/fail、
 失败原因和关键轨迹，作为后续优化流水线的基准线。
@@ -18,6 +18,7 @@
 import json
 import time
 from dataclasses import dataclass, field
+import hashlib
 from pathlib import Path
 from typing import Optional
 
@@ -331,7 +332,7 @@ async def _run_real_split(
         gt_items = []
         for case in cases_data:
             gt_items.append({
-                "id": hash(case["case_id"]) % 10000,
+                "id": int(hashlib.sha256(case["case_id"].encode()).hexdigest()[:8], 16) % 10000 + 1,
                 "image": f"eval/dataset/test_plates/{case['image']}",
                 "plate_number": case["ground_truth"],
                 "conditions": case.get("conditions", {}),
@@ -351,9 +352,12 @@ async def _run_real_split(
         report = await evaluator.run(verbose=False)
 
         # 转换为 BaselineCaseResult 列表
+        # fix: use path-based mapping instead of image_id positional lookup
+        image_to_case = {c.get("image", ""): c["case_id"] for c in cases_data}
         case_results: list[BaselineCaseResult] = []
         for r in report.details:
-            case_id = cases_data[r.image_id - 1]["case_id"] if r.image_id <= len(cases_data) else f"case_{r.image_id}"
+            image_key = Path(r.image_path).name if r.image_path else ""
+            case_id = image_to_case.get(image_key, image_to_case.get(r.image_path, f"case_{getattr(r, 'image_id', '?')}" ))
             case_result = BaselineCaseResult(
                 case_id=case_id,
                 image=r.image_path,