feat(eval): frozen-task ablation benchmark — and three guard defects it uncovered#389
Open
ceilf6 wants to merge 11 commits into
Open
feat(eval): frozen-task ablation benchmark — and three guard defects it uncovered#389ceilf6 wants to merge 11 commits into
ceilf6 wants to merge 11 commits into
Conversation
… targets with failing tests)
…ugfix / 5 refactor)
…iption Haiku, no API key)
…, fail fast on dead backend Smoke run exposed two silent-degradation paths: expired claude CLI auth produced zero-call 'completed' tasks, and balanced-mode overwrite approvals were denied without an approval handler.
- disable external RAG for both arms (default repo pulls unrelated wiki content into query context; noise + token cost) - query tasks get --files hints mirroring real CLI usage (deterministic query planning read a nonexistent README instead of target sources); two navigation tasks intentionally keep no hints - fixture gains a minimal README (structure only, no interface docs) Task set amended before the measured run; both arms use the identical set.
- disallow all tools + override system prompt: directory-listing style prompts triggered tool_use and error_max_turns under --max-turns 1 - run CLI from an empty cwd: Claude Code was injecting the host repo's CLAUDE.md and project context into every eval call (contamination + tokens) - include stdout head and prompt size in failure diagnostics
Frozen 30-task benchmark (27 runs per arm; 3 refactor tasks unrun in both arms due to quota, symmetrically excluded). Findings: - SDD on/off shows no measurable difference in first-pass rate (-3.7pp). - The hallucination guard recorded zero interceptions across both arms, yet syntactically invalid files (markdown fences in .tsx, TS1127) landed on disk. Three root causes located and documented in the report.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
代码评审报告: feat(eval): frozen-task ablation benchmark
风险等级: 中
处理建议: 请求修改
决策摘要: 运行时改动本身是低风险的加法式变更，但 benchmark 夹具里植入的 bug fixture (clamp.ts) 与其自带测试逻辑冲突——测试断言上界返回 max，而 fixture 有意返回 min；这会让 fixture 的基线测试永远红，需在合并前澄清意图。
级联分析
- 变更符号:
runFrontAgentTask（新增可选hallucinationGuard透传）、benchmark harness（新增，独立于运行时） - 受影响流程:
run.ts两处AgentConfig组装点；benchmark orchestration 为独立入口，不进入生产路径 - 变更集外调用方:
runFrontAgentTask报告 65 下游节点，但改动为undefined默认的加法字段，既有调用方行为不变；新测试覆盖 "leaves hallucinationGuard undefined" - 置信度: medium（
run.ts具体 diff 被截断，未能逐行确认两处组装点；结论依赖 PR 描述与 meta.json）
问题发现
-
[中] fixture
clamp.ts的植入 bug 与其自带测试自相矛盾- 证据:
clamp.ts:3在value > max时return min（植入的 bug），而clamp.test.ts:9断言clamp(11, 0, 10)toBe10。fixture 测试对未修复的 clamp 直接失败。 - 受影响调用方/流程:
bugfix-clamp任务的基线；若 harness 在 agent 修复前跑该测试作为基线校验，会误判环境状态。 - 最小可行修复: 明确这是"待修复 bug 的失败测试"（预期红），并在文档或 fixture README 中标注哪些 test 是 planted-failing baseline，避免与真正的环境损坏混淆。
- 证据:
-
[低]
hallucinationGuard透传只让 guard "addressable"，但据关联 issue #386 executor 路径仍不消费enabledChecks
行级发现
- [benchmarks/eval/fixture/src/utils/clamp.ts:3]
value > max时返回min是植入 bug；请确认这是bugfix-clamp的故意缺陷，并标注对应clamp.test.ts为预期失败基线，否则 harness 基线校验会把它当环境故障。 - [benchmarks/eval/run-eval.mjs:110]
pass = !runError && checkResults.length > 0 && checkResults.every(c => c.ok)：空 checks 数组会判 FAIL，但query-structure等任务只有result_containschecks，逻辑正确；仅提示当任务漏配 checks 时会静默计入 FAIL 而非报错，建议对checks.length === 0显式抛错以区分"漏配"与"真失败"。 - [benchmarks/eval/run-eval.mjs:31]
sddPath: 'sdd.disabled.yaml'依赖"指向不存在文件→existsSync 判无 SDD"的隐式约定；若某天 fixture 里恰好出现同名文件，ablation arm 会静默启用 SDD。建议改用显式sdd: false/disableSdd开关而非依赖文件缺失。
Karpathy 评审
- 假设: harness 依赖多处隐式约定——ablation 通过"不存在的 sddPath"关闭 SDD、
llmCalls === 0即判后端损坏、fixture 植入 bug 的测试预期为红。这些假设未在代码内显式标注，后续维护者易踩坑。 - 简洁性: harness 结构清晰、单一入口，无过度抽象；
checks.mjs的 switch 分发恰当。 - 结构质量: 新增均为独立 benchmark 目录，未污染生产路径；运行时改动是最小加法式透传，符合外科手术式变更。
- 变更范围: 38 文件中绝大多数是 fixture / lockfile / 结果数据，属 benchmark 自洽资产；
meta.json是索引产物，噪音大但无害。运行时真实改动面很小。 - 验证:
run.ts加了 2 个新测试，workflow 加 1 个契约测试，声称全绿；但run.tsdiff 被截断，无法逐行确认两处AgentConfig组装点都正确透传。
缺失覆盖
run.ts的hallucinationGuard透传：需确认两个组装点都有测试，而不仅是 "undefined 默认" 一例；应补一个"caller 传入 guard config → AgentConfig 两处都收到"的断言（diff 截断，无法核实是否已覆盖）。- fixture planted-bug 测试的基线状态：建议补一个 harness 层的自检，区分"植入失败测试"与"环境损坏"，否则
bugfix-clamp等任务的初始红态难以与真实故障区分。 - ablation arm SDD 关闭的正向验证：应有测试断言
sdd.disabled.yaml路径确实导致 SDD 未加载，防止隐式约定被未来改动破坏。
| @@ -0,0 +1,5 @@ | |||
| export function clamp(value: number, min: number, max: number): number { | |||
| if (value < min) return min; | |||
| if (value > max) return min; | |||
There was a problem hiding this comment.
value > max 时返回 min 是植入 bug；请确认这是 bugfix-clamp 的故意缺陷，并标注对应 clamp.test.ts 为预期失败基线，否则 harness 基线校验会把它当环境故障。
| const usageAfter = getUsageTally(); | ||
| const checkResults = runError ? [] : runChecks(task.checks, { workspace: ws, resultText }); | ||
| const pass = !runError && checkResults.length > 0 && checkResults.every((c) => c.ok); | ||
| const llmCalls = usageAfter.calls - usageBefore.calls; |
There was a problem hiding this comment.
pass = !runError && checkResults.length > 0 && checkResults.every(c => c.ok)：空 checks 数组会判 FAIL，但 query-structure 等任务只有 result_contains checks，逻辑正确；仅提示当任务漏配 checks 时会静默计入 FAIL 而非报错，建议对 checks.length === 0 显式抛错以区分"漏配"与"真失败"。
| ablation: { | ||
| // 指向不存在的文件 → run.ts existsSync 判定为无 SDD | ||
| sddPath: 'sdd.disabled.yaml', | ||
| hallucinationGuard: { |
There was a problem hiding this comment.
sddPath: 'sdd.disabled.yaml' 依赖"指向不存在文件→existsSync 判无 SDD"的隐式约定；若某天 fixture 里恰好出现同名文件，ablation arm 会静默启用 SDD。建议改用显式 sdd: false/disableSdd 开关而非依赖文件缺失。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A reproducible ablation benchmark for the agent's own architecture claims — and the defects it surfaced.
Harness (
benchmarks/eval/)tsc --noEmit,vitest, file existence/content assertions. No human grading.runtime-node:hallucinationGuardconfig is now passed through intoAgentConfig(it was silently dropped), so the guard is at least addressable from the runtime API.Results (
benchmarks/results/2026-07-12-sdd-ablation.md— 27 runs per arm).tsx,TS1127) still landed on disk.Defects found (filed as issues)
enabledChecksis dead config on the executor path: the executor callsvalidateFilePath/validateCode, which never consult it. The guard cannot be disabled, which also voided the guard arm of this very benchmark.step.validation.some(v => v.required), and LLM plans routinely emit an empty array) → invalid files persist.validation_failednever fires, so guard effectiveness is unobservable in telemetry.Merging with the negative results intact is deliberate. Fixing #386–#388 and re-running the same frozen set is the follow-up.
GitNexus Impact Summary
packages/runtime-node/src/run.ts(mcp-boundary) adds one optional fieldhallucinationGuard, forwarded intoAgentConfigat both assembly sites;README.md(authority-docs) documents the benchmark.detect_changesreports no uncommitted diff;impactonrunFrontAgentTaskreports 65 downstream nodes (direct 30, processes 6, modules 8). The change is additive with anundefineddefault, so no existing caller behavior changes — covered by the new test "leaves hallucinationGuard undefined when the caller omits it".pnpm build;vitest packages/runtime-node/src/run.test.ts(12 pass, 2 new);npm run test:workflows(33 pass, 1 new contract test);biome checkclean;contract:localpassed.Sample completeness
3 refactor tasks (
refactor-extract-listitem,refactor-app-title,refactor-button-memo) were not executed in either arm (quota), and are symmetrically excluded. All conclusions rest on the 27 tasks common to both arms.🤖 Generated with Claude Code