Skip to content

Commit 5459223

Browse files
committed
docs(eval): add Chinese design explanation
1 parent e48ca2a commit 5459223

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

  • examples/optimization/eval_optimize_loop
Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
1-
# Design notes
1+
# 方案设计说明
22

3-
The loop has four boundaries. Evaluation normalizes SDK results into stable case snapshots. Attribution reads actual intermediate trace data first and uses expected conversation data only as a reference. Candidate generation is either the checked-in fixture backend or a thin live adapter around the public `AgentOptimizer` API. Acceptance is owned by the Gate, never by an optimizer round status or summary.
3+
本示例在公开的 `AgentEvaluator``AgentOptimizer` 之上增加独立编排层,形成“基线评测、失败归因、候选生成、独立回归、Gate 决策、审计落盘”的完整闭环,不修改 SDK 公共 API。失败归因采用规则优先策略:先根据执行异常、实际与期望工具选择、参数差异、工具响应、JSON 格式和判定原因分类;trace 模式优先使用真实中间轨迹,期望会话只作为参照。只有规则不能解释时才保留可扩展的 judge 入口;每个结论都带有类型、证据和置信度。
44

5-
`PromptSandbox` isolates every candidate: source prompt files stay untouched during evaluation. The live adapter builds a temporary target, passes an SDK-only runtime configuration with environment placeholders intact, uses `update_source=False` and `verbose=0`, and archives optimizer artifacts. It extracts complete prompt maps only, de-duplicates their canonical digest, then independently runs full train and validation evaluation for each proposal. Optional write-back remains off by default and uses a baseline/candidate digest check after Gate success.
5+
接受策略由 Gate 独立掌握,优化器的 round 状态或摘要不能直接决定通过。候选必须具有完整提示词映射,并在 `PromptSandbox` 中重新跑完整训练集和验证集。Gate 会拒绝评测不完整、新增 hard fail、关键回归、超出允许范围的回归、指标低于下限、验证集无增益、成本或耗时超预算以及违反平局策略的候选;未知成本会记录为警告。赢家按新增 hard fail、关键回归数、验证集通过率和得分、成本、耗时与稳定候选 ID 排序。
66

7-
The Gate rejects incomplete evidence, unavailable validation deltas, new hard failures, critical or excessive regressions, metric-floor misses, train-up/validation-down overfit, configured generalization-gap, cost, duration, and tie-policy violations. Unknown cost is recorded as a warning. It ranks only independently evaluated and accepted candidates by hard failures, critical regressions, validation pass rate, validation score, cost, duration, and candidate id, making selection stable.
7+
防过拟合依靠训练集与验证集的独立全量回归,而不是读取优化器轮次指标。若训练提升而验证下降,或泛化差距超过配置阈值,Gate 会直接拒绝。优化期间固定 `update_source=False`,候选只作用于临时目标提示词目录;默认不回写,显式允许回写时还要校验基线和候选摘要。
88

9-
Every path writes a Pydantic-readable `OptimizationReport` plus secret-free input, environment, raw, normalized, candidate, and Gate artifacts. Fake and trace runs are deterministic and suitable for regression tests. Existing third-party `LangChainPendingDeprecationWarning` and local `requests` dependency-compatibility warnings may appear during test execution; they are external warnings rather than pipeline decisions.
9+
每次运行都会生成可由 Pydantic 读取的 `OptimizationReport`,并保存脱敏后的输入、环境、原始、归一化、候选与 Gate 审计产物。fake 和 trace 路径不需要 API Key、网络或 LLM judge,且输出稳定可复现;live 模式缺少必要环境变量会在启动优化器前安全退出。

0 commit comments

Comments
 (0)