feat(agent-eval): probe A/B harness and log parser (PR 9)

.github/workflows/ci.yml

@@ -95,6 +95,9 @@ jobs:
       - name: Golden query regression (fixtures/minimal)
         run: bun run test:golden
 
+      - name: Agent eval probe harness (fixtures/minimal)
+        run: bun run test:agent-eval
+
   build:
     name: 🧰 Build
     needs: skip-ci

.gitignore

@@ -15,3 +15,4 @@ fixtures/golden/scenarios.external.json
 # QA chat prompts tied to a private/local index (paths + product names)
 fixtures/qa/*.local.md
 fixtures/benchmark/*.local.json
+.agent-eval/

docs/benchmark.md

@@ -10,6 +10,7 @@
 | **Measure SQL vs glob+read+regex** after an index exists — `src/benchmark.ts`, scenarios, fixtures                                                       | [§ The benchmark script](#the-benchmark-script)                                  |
 | **Compare `codemap query` table vs `--json` stdout** (lines/bytes) on an existing index                                                                  | [§ Query stdout (`benchmark:query`)](#query-stdout-table-vs-json-benchmarkquery) |
 | **Guardrail full-rebuild per-phase walls against a committed baseline** (local + weekly scheduled)                                                       | [§ Perf baseline (regression guardrail)](#perf-baseline-regression-guardrail)    |
+| **A/B agent eval** — indexed MCP-on vs file-scan MCP-off tool-call + token comparison on fixed probes                                                    | [§ Agent eval harness](#agent-eval-harness)                                      |
 
 ---
 
@@ -282,7 +283,31 @@ bun run dev --full
 bun run benchmark
 ```
 
-**CI:** the workflow **Benchmark (fixture)** runs the same steps with `CODEMAP_ROOT=$GITHUB_WORKSPACE/fixtures/minimal`.
+**CI:** the **Test** job runs `bun run test:agent-eval` (~50s probe smoke on `fixtures/minimal`); **Benchmark (fixture)** indexes the same corpus and runs `bun run benchmark`.
+
+### Agent eval harness
+
+Dev-only A/B harness in [`scripts/agent-eval/`](../scripts/agent-eval/) (not shipped in npm). Indexes the fixture corpus once, then compares a **codemap MCP-on** arm (one `query` tool call per probe) against an **MCP-off** arm that simulates agent discovery without the index (`glob` → `read` × N → `grep`). Probe **prompts and SQL/recipe** reuse [golden scenarios](../fixtures/golden/scenarios.json) via `goldenId`; the MCP-off **traditional** regex/globs in [`scripts/agent-eval/scenarios.json`](../scripts/agent-eval/scenarios.json) approximate naive file discovery (not byte-identical to golden SQL).
+
+**One-command local run:**
+
+```bash
+bash scripts/agent-eval/run-arms.sh
+# default output: .agent-eval/comparison.json
+# exits non-zero when any probe's scenarioSuccess is false
+```
+
+Environment overrides: `AGENT_EVAL_OUTPUT`, `AGENT_EVAL_FIXTURE_ROOT`. **`AGENT_EVAL_RUNS`** (or `--runs`) repeats each probe and **averages** `wallMs`, `estTokens`, `resultCount`, and `toolCallCount` (rounded); `toolSequence` stays from the first run. Optional real agent session logs: `AGENT_EVAL_LOG=path/to/export.json bash scripts/agent-eval/run-arms.sh` (prints parsed tool metrics via `print-log-metrics.ts`).
+
+**Metrics (per scenario and summary):** tool-call sequence + count, wall time, estimated tokens (`chars / 4` on prompt + payload — MCP-on includes SQL + JSON rows; MCP-off includes bytes read + grep hits), per-arm `success` (non-empty results) plus `scenarioSuccess` when both arms succeed. Results stay local JSON — no telemetry upload ([plan](./plans/agent-eval-harness.md) L.5).
+
+**Methodology notes:**
+
+- **Probe mode** is deterministic (no LLM): it measures structural cost of indexed SQL vs traditional file scan on the same corpus. Use it for regression guardrails and fixture tuning.
+- **Log mode** parses exported agent transcripts (entries / messages / line formats) when you run live A/B sessions with MCP on vs off.
+- External public repos (zod, fastify, etc.) are a follow-up: point `AGENT_EVAL_FIXTURE_ROOT` at an indexed tree and extend probe definitions — same harness, not duplicated fixtures.
+
+Plan: [`docs/plans/agent-eval-harness.md`](./plans/agent-eval-harness.md). PR CI runs `bun run test:agent-eval` in the **Test** job (~50s smoke on `fixtures/minimal`); optional nightly / `workflow_dispatch` for external fixtures is not wired yet.
 
 **Correctness (golden queries):** `bun run test:golden` indexes `fixtures/minimal`, runs SQL against [fixtures/golden/scenarios.json](../fixtures/golden/scenarios.json), and compares to [fixtures/golden/minimal/](../fixtures/golden/minimal/). See [golden-queries.md](./golden-queries.md). Refresh goldens after intentional fixture or schema changes: `bun scripts/query-golden.ts --update`.
 

docs/plans/agent-eval-harness.md

@@ -1,8 +1,8 @@
 # Agent eval harness — plan
 
-> **Status:** open · **Priority:** P1 · **Effort:** M (~2 weeks)
+> **Status:** open (probe tracer bullet shipped [#139](https://github.com/stainless-code/codemap/pull/139)) · **Priority:** P1 · **Effort:** M (~2 weeks)
 >
-> **Motivator:** Codemap claims agent-discovery wins ([why-codemap.md](../why-codemap.md), [benchmark.md](../benchmark.md)) but CI only gates query latency and golden SQL. Need falsifiable A/B: agent with MCP vs without, measuring tool-call count and tokens on fixed tasks.
+> **Motivator:** Codemap claims agent-discovery wins ([why-codemap.md](../why-codemap.md), [benchmark.md](../benchmark.md)). PR CI now gates golden SQL plus deterministic probe A/B smoke on `fixtures/minimal`; still need falsifiable **live** agent runs (MCP on vs off) and external-fixture benchmarks measuring tool-call count and tokens on fixed tasks.
 >
 > **Roadmap:** [§ Backlog](../roadmap.md#backlog) (falsifiable benchmark item) · [agent-surface-and-ops § P1](./agent-surface-and-ops.md#p1)
 
@@ -22,20 +22,24 @@
 
 ## Implementation steps
 
-1. **`scripts/agent-eval/run-arms.sh`** — orchestrate N runs per arm
-2. **Probe scripts** — structured tasks mirroring golden queries ("find symbol X", "who imports Y", "fan-in top 10")
-3. **Parser for agent logs** — extract tool names from Claude/Cursor export format (start with one agent)
+1. **`scripts/agent-eval/run-arms.sh`** — orchestrate N runs per arm ✅
+2. **Probe scripts** — structured tasks mirroring golden queries ("find symbol X", "who imports Y", "fan-in top 10") ✅ (3 probes shipped)
+3. **Parser for agent logs** — extract tool names from Claude/Cursor export format ✅
 4. **Summary reporter** — markdown table for docs/benchmark.md
-5. **CI job** — optional nightly or manual `workflow_dispatch` (public fixtures only)
-6. **Link to [mcp-tool-allowlist](./mcp-tool-allowlist.md)** for minimal-tool arms
+5. **PR CI probe gate** — `test:agent-eval` in Test job on `fixtures/minimal` ✅
+6. **Optional nightly / `workflow_dispatch`** — public external fixtures only
+7. **Link to [mcp-tool-allowlist](./mcp-tool-allowlist.md)** for minimal-tool live arms
 
 ---
 
 ## Acceptance
 
-- [ ] One-command local run produces comparison JSON
-- [ ] At least 3 scenarios covered
-- [ ] Documented methodology section in benchmark.md
+- [x] One-command local run produces comparison JSON (`bash scripts/agent-eval/run-arms.sh`)
+- [x] At least 3 scenarios covered (`scripts/agent-eval/scenarios.json`)
+- [x] Documented methodology section in benchmark.md
+- [x] PR CI gates probe harness (`bun run test:agent-eval` in Test job on `fixtures/minimal`)
+- [ ] Optional nightly or manual `workflow_dispatch` (public external fixtures only)
+- [ ] Live agent A/B arms with MCP allowlist subset
 
 ---
 

docs/plans/agent-surface-delivery.md

@@ -10,11 +10,11 @@
 
 ## Quick resume
 
-| Next action                 | Detail                                                                                     |
-| --------------------------- | ------------------------------------------------------------------------------------------ |
-| **In review (merge-ready)** | [#138](https://github.com/stainless-code/codemap/pull/138) — field-qualified search (PR 7) |
-| **Start next**              | **PR 9** — `agent-eval-harness` (last P1) after #138 merges                                |
-| **Recently merged**         | [#135](https://github.com/stainless-code/codemap/pull/135) — agents init `--mcp` (PR 8)    |
+| Next action         | Detail                                                                                                             |
+| ------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| **Merge-ready**     | [#139](https://github.com/stainless-code/codemap/pull/139) — agent-eval harness (PR 9); CI green, review gate only |
+| **Recently merged** | [#138](https://github.com/stainless-code/codemap/pull/138) — field-qualified search (PR 7)                         |
+|                     | [#135](https://github.com/stainless-code/codemap/pull/135) — agents init `--mcp` (PR 8)                            |
 
 Update the table below when a PR merges or a new branch opens.
 
@@ -42,9 +42,9 @@ Max **3 parallel tracks** at once.
 | **5** | [`affected-tests-recipe`](./affected-tests-recipe.md) (+ Phase 2 MCP `affected` in [#133](https://github.com/stainless-code/codemap/pull/133)) | merged | [#132](https://github.com/stainless-code/codemap/pull/132), [#133](https://github.com/stainless-code/codemap/pull/133) | 3, 4          |
 
 | **6** | MCP half of trace (`trace` / `explore` / `node` tools) + update instructions | merged | [#134](https://github.com/stainless-code/codemap/pull/134) | PR 1, PR 4 |
-| **7** | [`field-qualified-search`](./field-qualified-search.md) | merge-ready | [#138](https://github.com/stainless-code/codemap/pull/138) | PR 1, PR 6 |
+| **7** | [`field-qualified-search`](./field-qualified-search.md) | merged | [#138](https://github.com/stainless-code/codemap/pull/138) | PR 1, PR 6 |
 | **8** | [`agents-init-mcp-wiring`](./agents-init-mcp-wiring.md) | merged | [#135](https://github.com/stainless-code/codemap/pull/135) | 3–5 |
-| **9** | [`agent-eval-harness`](./agent-eval-harness.md) | planned | PR 1, PR 8 (merged), allowlist | **last P1** — start after PR 7 or in parallel |
+| **9** | [`agent-eval-harness`](./agent-eval-harness.md) | open | [#139](https://github.com/stainless-code/codemap/pull/139) | PR 1, PR 8 (merged), allowlist |
 
 **Parallelization constraints**
 

docs/plans/mcp-tool-allowlist.md

@@ -24,7 +24,7 @@
 2. **Filter `register*Tool` invocations** — skip unlisted tools
 3. **Log registered set** on stderr at debug level or when allowlist active
 4. **Tests** — `CODEMAP_MCP_TOOLS=query,show` registers exactly two tools
-5. **Docs** — README env table; [agent-eval-harness](./agent-eval-harness.md) uses allowlist for arms
+5. **Docs** — README env table; live agent-eval arms may use allowlist (future — see [agent-eval-harness](./agent-eval-harness.md) open acceptance items)
 
 ---
 

docs/roadmap.md

@@ -71,7 +71,7 @@ Prioritized agent & indexing ops queue (2026-05). Index: [`plans/agent-surface-a
 - [ ] **Index lock + error log** — cross-process lock, `unlock`, `errors.log`. Plan: [`plans/index-lock-and-error-log.md`](./plans/index-lock-and-error-log.md). Effort: M.
 - [ ] **Parse worker hardening** — per-file timeout + worker recycle. Plan: [`plans/parse-worker-hardening.md`](./plans/parse-worker-hardening.md). Effort: M.
 - [x] **Field-qualified search** — `kind:` / `path:` / `name:` → SQL. Plan: [`plans/field-qualified-search.md`](./plans/field-qualified-search.md). [#138](https://github.com/stainless-code/codemap/pull/138). Effort: M.
-- [ ] **Agent eval harness** — A/B MCP tool-call + token metrics. Plan: [`plans/agent-eval-harness.md`](./plans/agent-eval-harness.md). Effort: M.
+- [ ] **Agent eval harness** — probe-mode A/B + PR CI gate shipped ([#139](https://github.com/stainless-code/codemap/pull/139)); live MCP arms + optional external nightly CI still open. Plan: [`plans/agent-eval-harness.md`](./plans/agent-eval-harness.md). Effort: M.
 
 **P2 — strategic (trigger-gated where noted)**
 

fixtures/agent-eval/sample-cursor-log.json

@@ -0,0 +1,23 @@
+{
+  "entries": [
+    {
+      "kind": "user",
+      "text": "Where is the usePermissions symbol defined?"
+    },
+    {
+      "kind": "tool_call",
+      "tool": "mcp_codemap_query",
+      "args": {
+        "sql": "SELECT name, kind, file_path FROM symbols WHERE name = 'usePermissions'"
+      }
+    },
+    {
+      "kind": "tool_result",
+      "tool": "mcp_codemap_query"
+    },
+    {
+      "kind": "assistant",
+      "text": "usePermissions is defined in src/hooks/usePermissions.ts"
+    }
+  ]
+}

package.json

@@ -46,7 +46,7 @@
     "benchmark:query": "bun scripts/benchmark-query-output.ts",
     "build": "tsdown",
     "changeset": "changeset",
-    "check": "bun run build && bun run --parallel format:check lint:ci test typecheck && bun run test:golden",
+    "check": "bun run build && bun run --parallel format:check lint:ci test typecheck && bun run test:golden && bun run test:agent-eval",
     "check-updates": "bun update -i --latest",
     "check:perf-baseline": "bun scripts/check-perf-baseline.ts",
     "check:perf-baseline:update": "bun scripts/check-perf-baseline.ts --update",
@@ -65,6 +65,7 @@
     "qa:external": "bun scripts/qa-external-repo.ts",
     "release": "changeset publish",
     "test": "bun test ./src",
+    "test:agent-eval": "bun test scripts/agent-eval",
     "test:ci": "bun run test:coverage",
     "test:coverage": "bun test --coverage ./src",
     "test:golden": "bun scripts/query-golden.ts",

scripts/agent-eval/metrics.ts

@@ -0,0 +1,8 @@
+/** Chars / 4 token estimate per agent-eval-harness plan L.4. */
+export function estimateTokens(chars: number): number {
+  return Math.ceil(chars / 4);
+}
+
+export function jsonCharLength(value: unknown): number {
+  return Buffer.byteLength(JSON.stringify(value), "utf-8");
+}