You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Numbers are stable for a given fixture + schema; re-run locally after intentional schema or probe changes.
369
369
370
-
### Harden-pr workflow eval
371
-
372
-
Manual A/B harness for [`.agents/skills/harden-pr/`](../.agents/skills/harden-pr/SKILL.md) — complements codemap **query** probes above. Measures whether an agent **detects, fixes, and verifies** production-bar gaps on a branch, not SQL vs grep cost.
| Autonomy | Zero mid-loop commit/babysit/next-pass prompts |
406
-
| Passes | ≤ mode cap (lite 2, full 3) |
407
-
408
-
**A/B skill versions:** same probe dir, two sessions (before/after skill edit); compare recall, false fixes, passes, autonomy.
409
-
410
-
Future: log parser for `Task` reviewer JSON (same spirit as `parse-agent-log.ts`) to automate recall scoring.
411
-
412
370
#### Dual-agent study (codemap self-index, provisional)
413
371
414
372
Exploratory runs on the **codemap repo** index (not `fixtures/minimal`) — four structural tasks, MCP-on vs MCP-forbidden (grep/read/shell only). Not pinned in CI; methodology caveats apply.
Copy file name to clipboardExpand all lines: docs/testing-coverage.md
-13Lines changed: 0 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,6 @@
14
14
|**In-repo test bench**|`bun run test:golden`| Index `fixtures/minimal/` (bench corpus) → compare `fixtures/golden/minimal/*.json`. Map: `fixtures/CAPABILITIES.json`. |
15
15
|**Golden guard**|`bun run test:scripts`|`scripts/query-golden-coverage-matrix.test.mjs` — every bundled recipe + substrate table has a scenario. |
16
16
|**Agent eval**|`bun run test:agent-eval`| Probe arms vs golden ids (MCP-on vs glob/read). |
17
-
|**Harden-pr probes**|`bun run test:harden-probes`| Schema + pre-fix fail + score oracle; manual agent run + `acceptance.sh` after fix. |
|**Apply CLI e2e**|`src/cli/cmd-apply.test.ts`| Temp project + full index: recipe dry-run/apply, `--rows`, second recipe disk apply. |
@@ -103,18 +102,6 @@ Codemap development uses **only** the committed bench ([fixtures/README.md](../f
103
102
104
103
`scripts/agent-eval/scenarios.json` — one probe per `CAPABILITIES.json` capability id (groups with `unitTests` or `enforcedBy` instead: `recipes.bundled`, `cli.bench-smoke`, `cli.mcp.http`). Each probe’s `goldenId` must appear in that group’s `goldenScenarios`; multiple substrate ids per group are fine — one matching probe is enough. Enforced by `scripts/agent-eval/capability-probes.test.mjs` in `test:scripts`.
105
104
106
-
### Harden-pr workflow probes
107
-
108
-
Manual eval for [`.agents/skills/harden-pr/`](../.agents/skills/harden-pr/SKILL.md) — not automated agent runs in CI yet.
|[`scripts/agent-eval/harden-scenarios.json`](../scripts/agent-eval/harden-scenarios.json)| Scenario index (probe dir, mode, acceptance script) |
114
-
|[`scripts/harden-probes/validate-fixtures.test.mjs`](../scripts/harden-probes/validate-fixtures.test.mjs)| Schema guard in `test:scripts`|
115
-
116
-
Run a probe: open the probe dir in Cursor → `/harden-pr lite` → score findings with `score-probe.mjs` → `bash acceptance.sh`. Mechanical smoke: `bun run test:harden-probes`. Full protocol: [benchmark.md § Harden-pr workflow eval](./benchmark.md#harden-pr-workflow-eval).
0 commit comments