Skip to content

Commit 8c6943c

Browse files
committed
chore(agents): drop harden-pr eval probes after validation
Keep harden-pr skill, LEDGER.md, and tracer-bullets wiring; remove fixtures/harden-probes, score-probe harness, and related docs.
1 parent 22b35a1 commit 8c6943c

15 files changed

Lines changed: 0 additions & 346 deletions

File tree

.agents/skills/harden-pr/SKILL.md

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,4 @@ Skill invocation **is** the commit authorization. After the loop: if fixes exist
294294
| Branch done | `/harden-pr full` or "production-ready pass" |
295295
| Deferred backlog | `/harden-pr reconcile` |
296296

297-
**Eval probes:** [`fixtures/harden-probes/`](../../../fixtures/harden-probes/) — manual workflow scoring; schema enforced by `scripts/harden-probes/validate-fixtures.test.mjs`.
298-
299297
Replaces the old copy-paste: _"spawn subagents → fix → loop until clean"_ — this skill **is** that loop.

docs/benchmark.md

Lines changed: 0 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -367,48 +367,6 @@ Environment overrides: `AGENT_EVAL_OUTPUT`, `AGENT_EVAL_FIXTURE_ROOT`, `AGENT_EV
367367

368368
Numbers are stable for a given fixture + schema; re-run locally after intentional schema or probe changes.
369369

370-
### Harden-pr workflow eval
371-
372-
Manual A/B harness for [`.agents/skills/harden-pr/`](../.agents/skills/harden-pr/SKILL.md) — complements codemap **query** probes above. Measures whether an agent **detects, fixes, and verifies** production-bar gaps on a branch, not SQL vs grep cost.
373-
374-
| Artifact | Role |
375-
| --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
376-
| [`fixtures/harden-probes/`](../fixtures/harden-probes/) | Mini corpora with injected gaps + `expected-findings.json` oracle |
377-
| [`scripts/agent-eval/harden-scenarios.json`](../scripts/agent-eval/harden-scenarios.json) | Scenario index (probe dir, mode, acceptance script) |
378-
| [`.agents/skills/harden-pr/LEDGER.md`](../.agents/skills/harden-pr/LEDGER.md) | Rejections (vet) + deferred backlog (reconcile) — one file |
379-
| [`scripts/harden-probes/validate-fixtures.test.mjs`](../scripts/harden-probes/validate-fixtures.test.mjs) | Golden finding schema guard (`test:scripts`) |
380-
381-
**Run one probe:**
382-
383-
```bash
384-
cd fixtures/harden-probes/missing-test
385-
# Cursor: attach harden-pr → `/harden-pr lite`
386-
bash acceptance.sh # after harden — must exit 0
387-
```
388-
389-
**Mechanical smoke (CI via `test:scripts`):**
390-
391-
```bash
392-
bun run test:harden-probes
393-
# schema guard + pre-fix acceptance fails + score-probe oracle
394-
395-
# Score agent findings JSON against golden:
396-
bun scripts/harden-probes/score-probe.mjs fixtures/harden-probes/missing-test .agent-eval/my-findings.json
397-
```
398-
399-
**Score each run** (spreadsheet or research note — not CI-gated yet):
400-
401-
| Metric | Pass |
402-
| -------- | ------------------------------------------------------------------------------ |
403-
| Recall | Finds ≥1 golden row in `expected-findings.json` (same `file` + production bar) |
404-
| Fix | `acceptance.sh` green; intent anchor unchanged |
405-
| Autonomy | Zero mid-loop commit/babysit/next-pass prompts |
406-
| Passes | ≤ mode cap (lite 2, full 3) |
407-
408-
**A/B skill versions:** same probe dir, two sessions (before/after skill edit); compare recall, false fixes, passes, autonomy.
409-
410-
Future: log parser for `Task` reviewer JSON (same spirit as `parse-agent-log.ts`) to automate recall scoring.
411-
412370
#### Dual-agent study (codemap self-index, provisional)
413371

414372
Exploratory runs on the **codemap repo** index (not `fixtures/minimal`) — four structural tasks, MCP-on vs MCP-forbidden (grep/read/shell only). Not pinned in CI; methodology caveats apply.

docs/testing-coverage.md

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,6 @@
1414
| **In-repo test bench** | `bun run test:golden` | Index `fixtures/minimal/` (bench corpus) → compare `fixtures/golden/minimal/*.json`. Map: `fixtures/CAPABILITIES.json`. |
1515
| **Golden guard** | `bun run test:scripts` | `scripts/query-golden-coverage-matrix.test.mjs` — every bundled recipe + substrate table has a scenario. |
1616
| **Agent eval** | `bun run test:agent-eval` | Probe arms vs golden ids (MCP-on vs glob/read). |
17-
| **Harden-pr probes** | `bun run test:harden-probes` | Schema + pre-fix fail + score oracle; manual agent run + `acceptance.sh` after fix. |
1817
| **Integration (git)** | `src/application/run-index.test.ts` | `runCodemapIndex` incremental paths: heritage + calls re-resolution, delete/reindex. |
1918
| **CLI e2e** | `src/cli/cmd-test-bench-e2e.test.ts`, `src/cli/cmd-cli-parity-e2e.test.ts` | Spawned CLI on `fixtures/minimal` (bench smoke + resource parity). |
2019
| **Apply CLI e2e** | `src/cli/cmd-apply.test.ts` | Temp project + full index: recipe dry-run/apply, `--rows`, second recipe disk apply. |
@@ -103,18 +102,6 @@ Codemap development uses **only** the committed bench ([fixtures/README.md](../f
103102

104103
`scripts/agent-eval/scenarios.json` — one probe per `CAPABILITIES.json` capability id (groups with `unitTests` or `enforcedBy` instead: `recipes.bundled`, `cli.bench-smoke`, `cli.mcp.http`). Each probe’s `goldenId` must appear in that group’s `goldenScenarios`; multiple substrate ids per group are fine — one matching probe is enough. Enforced by `scripts/agent-eval/capability-probes.test.mjs` in `test:scripts`.
105104

106-
### Harden-pr workflow probes
107-
108-
Manual eval for [`.agents/skills/harden-pr/`](../.agents/skills/harden-pr/SKILL.md) — not automated agent runs in CI yet.
109-
110-
| Artifact | Role |
111-
| --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
112-
| [`fixtures/harden-probes/`](../fixtures/harden-probes/) | Injected production-bar gaps + `expected-findings.json` oracle |
113-
| [`scripts/agent-eval/harden-scenarios.json`](../scripts/agent-eval/harden-scenarios.json) | Scenario index (probe dir, mode, acceptance script) |
114-
| [`scripts/harden-probes/validate-fixtures.test.mjs`](../scripts/harden-probes/validate-fixtures.test.mjs) | Schema guard in `test:scripts` |
115-
116-
Run a probe: open the probe dir in Cursor → `/harden-pr lite` → score findings with `score-probe.mjs``bash acceptance.sh`. Mechanical smoke: `bun run test:harden-probes`. Full protocol: [benchmark.md § Harden-pr workflow eval](./benchmark.md#harden-pr-workflow-eval).
117-
118105
---
119106

120107
## Fixture corpus

fixtures/harden-probes/README.md

Lines changed: 0 additions & 11 deletions
This file was deleted.

fixtures/harden-probes/missing-test/README.md

Lines changed: 0 additions & 26 deletions
This file was deleted.

fixtures/harden-probes/missing-test/acceptance.sh

Lines changed: 0 additions & 18 deletions
This file was deleted.

fixtures/harden-probes/missing-test/expected-findings.json

Lines changed: 0 additions & 12 deletions
This file was deleted.

fixtures/harden-probes/missing-test/package.json

Lines changed: 0 additions & 6 deletions
This file was deleted.

fixtures/harden-probes/missing-test/src/formatWidget.ts

Lines changed: 0 additions & 8 deletions
This file was deleted.

package.json

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,6 @@
7070
"test:coverage": "bun test --coverage ./src",
7171
"test:golden": "bun scripts/query-golden.ts",
7272
"test:golden:external": "bun scripts/query-golden.ts --corpus external",
73-
"test:harden-probes": "bash scripts/harden-probes/run-smoke.sh",
7473
"test:scripts": "bash -c 'files=$(find scripts -name \"*.test.mjs\"); if [ -z \"$files\" ]; then echo \"no scripts test files found\" >&2; exit 1; fi; exec bun test $files'",
7574
"typecheck": "tsgo --noEmit",
7675
"version": "changeset version && bun run format CHANGELOG.md"

0 commit comments

Comments
 (0)