Skip to content

Commit 4213898

Browse files
SutuSebastianSutuSebastian
committed
test: formalize in-repo test bench and capability manifest
Add fixtures/README.md, CAPABILITIES.json, and in-repo-test-bench plan so CI validates Codemap without external CODEMAP_ROOT. Bench corpus stays at fixtures/minimal; --corpus bench aliases minimal. Add method-call bench slice and golden; extend coverage-matrix guard for manifest files.
1 parent a4d4078 commit 4213898

24 files changed

Lines changed: 486 additions & 47 deletions

‎docs/golden-queries.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -52,11 +52,11 @@ We **do not** commit another product’s source tree, paths, business strings, o
5252

5353
## Tier model
5454

55-
| Tier | Corpus | When | Purpose |
56-
| --------------- | ---------------------------- | -------------------------- | ---------------------------------------- |
57-
| **A** | `fixtures/minimal` (in-repo) | Every PR / `bun run check` | Fast, **committed** goldens |
58-
| **B** | Local path via `CODEMAP_*` | Maintainer machine | Scale; goldens **optional / gitignored** |
59-
| **B′** (future) | Public OSS fixture only | CI optional | Larger committed corpus if license OK |
55+
| Tier | Corpus | When | Purpose |
56+
| -------------------------- | ---------------------------------------- | -------------------------- | --------------------------------------------------------------------------------------- |
57+
| **A (in-repo test bench)** | `fixtures/minimal` + `fixtures/golden/` | Every PR / `bun run check` | **Canonical** for Codemap development — see [fixtures/README.md](../fixtures/README.md) |
58+
| **B (consumer-only)** | Local path via `CODEMAP_*` | Private app validation | Goldens **gitignored** — not required to develop Codemap |
59+
| **Bench growth** | `fixtures/minimal` + `CAPABILITIES.json` | Ongoing | Plan: [plans/in-repo-test-bench.md](./plans/in-repo-test-bench.md) |
6060

6161
---
6262

‎docs/plans/in-repo-test-bench.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# In-repo test bench (no external corpus)
2+
3+
**Status:** Open — Phase 1 shipped on `feat/unresolved-calls-staging` follow-ups; Phases 2–4 backlog.
4+
**Effort:** M (ongoing; expand incrementally).
5+
**Supersedes for maintainers:** relying on Tier B `CODEMAP_ROOT` / private clones to validate Codemap itself.
6+
7+
## Problem
8+
9+
`fixtures/minimal` + `test:golden` already cover **all 58 bundled recipes** and most indexed tables, but the name “minimal” and Tier B docs imply a **second, external tree** is needed for realism. Contributors should have **one committed bench** inside this repo.
10+
11+
## Goal
12+
13+
| Audience | Outcome |
14+
| --------------- | ------------------------------------------------------------------------ |
15+
| **CI** | `bun run check` proves index + recipes + substrate without `CODEMAP_*` |
16+
| **Maintainers** | `fixtures/CAPABILITIES.json` maps capability → source files → golden ids |
17+
| **Consumers** | Tier B (`test:golden:external`) stays optional for _their_ apps only |
18+
19+
## Architecture
20+
21+
```text
22+
fixtures/
23+
README.md ← test bench entry (this plan)
24+
CAPABILITIES.json ← manifest (validated in test:scripts)
25+
minimal/ ← corpus (grow in place or add bench/* slices)
26+
golden/scenarios.json
27+
golden/minimal/*.json
28+
```
29+
30+
**Do not** add proprietary or gitignored source trees. **Do** add small, intentional files under `fixtures/minimal/` per capability gap.
31+
32+
## Phases
33+
34+
### Phase 1 — Document + guard (shipped)
35+
36+
- [x] `fixtures/README.md`, `fixtures/CAPABILITIES.json`
37+
- [x] `docs/testing-coverage.md`, `query-golden-coverage-matrix.test.mjs`
38+
- [x] Substrate pin-down goldens + `index-table-stats`
39+
- [x] Clarify Tier B = consumer-only in `golden-queries.md`
40+
41+
### Phase 2 — Corpus depth (next)
42+
43+
Target gaps from `knownGaps` in the manifest:
44+
45+
| Gap | Fixture slice | Golden |
46+
| ------------------------------- | --------------------------------------------------- | ------------------------------------------ |
47+
| Method-call resolution | `src/bench/method-call-sites.ts` | `calls-method-ping-unresolved` |
48+
| Qualified heritage / namespaces | extend `heritage-qualified.ts` or `src/bench/` | new SQL scenarios |
49+
| `affected-tests` param matrix | already partial — ensure all 6 scenarios stay green |
50+
| Project-local recipe | `.codemap/recipes/shop-symbols` | dedicated scenario (not only files-hashes) |
51+
52+
### Phase 3 — CLI / MCP bench pack
53+
54+
- [ ] `src/cli/cmd-test-bench-e2e.test.ts` — spawn CLI against `fixtures/minimal` for: `show`, `snippet`, `impact`, `trace`, `validate`, `context`, `query --format sarif` (smoke exit 0)
55+
- [ ] Expand `test:agent-eval` probes to cover one scenario per capability group in `CAPABILITIES.json`
56+
57+
### Phase 4 — Scale (optional, replaces Tier B′)
58+
59+
- [ ] If `minimal/` exceeds ~50 files, add `fixtures/bench/` as a **second committed corpus** with its own `scenarios.bench.json` — still in-repo, still CI
60+
- [ ] Or: rename `minimal` → `bench` in a breaking release with path alias `fixtures/minimal` → symlink
61+
62+
## Non-goals
63+
64+
- Cloning zod/fastify/etc. into the repo (roadmap backlog stays **local** for those benchmarks)
65+
- Replacing unit tests for parsers/CLI
66+
- Tier B removal — keep for consumer private validation
67+
68+
## Validation
69+
70+
```bash
71+
bun run test:scripts # CAPABILITIES.json ↔ scenarios.json
72+
bun run test:golden
73+
bun run check
74+
```
75+
76+
## References
77+
78+
- [testing-coverage.md](../testing-coverage.md)
79+
- [golden-queries.md](../golden-queries.md)
80+
- [fixtures/minimal/README.md](../../fixtures/minimal/README.md)

‎docs/roadmap.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ _(none — P1 agent-surface work complete)_
6767

6868
- [ ] **Framework route extraction** — Express / React Router / NestJS `http_routes` substrate. Plan: [`plans/framework-route-extraction.md`](./plans/framework-route-extraction.md). Blocked on C.9 contract. Effort: L.
6969
- [ ] **Callback dispatch synthesis** — heuristic `calls` with `provenance`. Plan: [`plans/callback-dispatch-synthesis.md`](./plans/callback-dispatch-synthesis.md). Effort: L.
70+
- [ ] **In-repo test bench depth** — grow `fixtures/minimal` + `CAPABILITIES.json` so all capability groups (CLI/MCP e2e, method-call slices, shop-symbols recipe) are covered without external `CODEMAP_ROOT`. Plan: [`plans/in-repo-test-bench.md`](./plans/in-repo-test-bench.md). Effort: M.
7071
- [ ] **Unresolved calls staging** — two-phase call resolution queue. Plan: [`plans/unresolved-calls-staging.md`](./plans/unresolved-calls-staging.md). Effort: L.
7172
- [ ] **Cross-project MCP root** — optional `root` on tools + DB cache. Plan: [`plans/cross-project-mcp-root.md`](./plans/cross-project-mcp-root.md). Effort: M.
7273
- [ ] **FTS default-on evaluation** — measure DB size tax; maybe flip default. Plan: [`plans/fts-default-on-evaluation.md`](./plans/fts-default-on-evaluation.md). Effort: S–M.

‎docs/testing-coverage.md‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,15 +8,15 @@
88

99
## Harness layers
1010

11-
| Layer | Command | What it proves |
12-
| --------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------- |
13-
| **Unit** | `bun test ./src` | Parsers, DB DDL, engines, CLI/MCP handlers in isolation (`:memory:` or mocks). |
14-
| **Golden (Tier A)** | `bun run test:golden` | Full index of `fixtures/minimal/` → SQL/recipe output vs `fixtures/golden/minimal/*.json`. |
15-
| **Golden guard** | `bun run test:scripts` | `scripts/query-golden-coverage-matrix.test.mjs` — every bundled recipe + substrate table has a scenario. |
16-
| **Agent eval** | `bun run test:agent-eval` | Probe arms vs golden ids (MCP-on vs glob/read). |
17-
| **Integration (git)** | `src/application/run-index.test.ts` | `runCodemapIndex` incremental paths: heritage + calls re-resolution, delete/reindex. |
18-
| **CLI e2e** | `cmd-cli-parity-e2e.test.ts` | Spawned CLI against `fixtures/minimal`. |
19-
| **Check** | `bun run check` | build + lint + unit + scripts + golden + agent-eval. |
11+
| Layer | Command | What it proves |
12+
| ---------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
13+
| **Unit** | `bun test ./src` | Parsers, DB DDL, engines, CLI/MCP handlers in isolation (`:memory:` or mocks). |
14+
| **In-repo test bench** | `bun run test:golden` | Index `fixtures/minimal/` (bench corpus) → compare `fixtures/golden/minimal/*.json`. Map: `fixtures/CAPABILITIES.json`. |
15+
| **Golden guard** | `bun run test:scripts` | `scripts/query-golden-coverage-matrix.test.mjs` — every bundled recipe + substrate table has a scenario. |
16+
| **Agent eval** | `bun run test:agent-eval` | Probe arms vs golden ids (MCP-on vs glob/read). |
17+
| **Integration (git)** | `src/application/run-index.test.ts` | `runCodemapIndex` incremental paths: heritage + calls re-resolution, delete/reindex. |
18+
| **CLI e2e** | `cmd-cli-parity-e2e.test.ts` | Spawned CLI against `fixtures/minimal`. |
19+
| **Check** | `bun run check` | build + lint + unit + scripts + golden + agent-eval. |
2020

2121
Refresh Tier A goldens after intentional fixture or schema changes:
2222

@@ -65,7 +65,11 @@ Core graph tables (`files`, `symbols`, `imports`, …) are covered by many exist
6565

6666
---
6767

68-
## Intentionally not in Tier A goldens
68+
## In-repo bench vs external trees
69+
70+
Codemap development uses **only** the committed bench ([fixtures/README.md](../fixtures/README.md)). `bun run test:golden:external` is for consumers testing against private apps — not a maintainer requirement.
71+
72+
## Intentionally not in bench goldens
6973

7074
| Surface | Why | Where tested |
7175
| ----------------- | -------------------- | ----------------------------------------------------------- |

‎fixtures/CAPABILITIES.json‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
{
2+
"version": 1,
3+
"corpusRoot": "fixtures/minimal",
4+
"goldenScenariosFile": "fixtures/golden/scenarios.json",
5+
"goldenDir": "fixtures/golden/minimal",
6+
"capabilities": [
7+
{
8+
"id": "index.schema",
9+
"description": "All indexed tables populated on full rebuild",
10+
"fixtureFiles": ["src/consumer.ts", "src/api/client.ts"],
11+
"goldenScenarios": ["index-table-stats", "index-summary"]
12+
},
13+
{
14+
"id": "symbols.imports.exports",
15+
"description": "Tier 1 symbols, imports, exports, dependencies",
16+
"fixtureFiles": [
17+
"src/consumer.ts",
18+
"src/api/client.ts",
19+
"src/utils/date.ts"
20+
],
21+
"goldenScenarios": [
22+
"symbol-usePermissions",
23+
"imports-consumer-alias",
24+
"dependencies-from-consumer",
25+
"exports-client"
26+
]
27+
},
28+
{
29+
"id": "substrate.references.bindings",
30+
"description": "Tier 2 references and bindings resolution",
31+
"fixtureFiles": [
32+
"src/components/shop/ProductCard.tsx",
33+
"src/consumer.ts",
34+
"src/components/shop/index.ts"
35+
],
36+
"goldenScenarios": [
37+
"references-product-card-perms",
38+
"bindings-createClient",
39+
"find-symbol-references",
40+
"find-re-exported-bindings"
41+
]
42+
},
43+
{
44+
"id": "calls.graph",
45+
"description": "Call edges, paths, optional chains, constructors",
46+
"fixtureFiles": ["src/consumer.ts", "src/api/client.ts"],
47+
"goldenScenarios": [
48+
"calls-consumer",
49+
"find-call-sites",
50+
"call-path",
51+
"calls-createClient-resolved"
52+
]
53+
},
54+
{
55+
"id": "calls.phase2",
56+
"description": "Two-phase call resolution and unresolved_calls queue",
57+
"fixtureFiles": [
58+
"src/consumer.ts",
59+
"src/lib/complexity-fixture.ts",
60+
"src/bench/method-call-sites.ts"
61+
],
62+
"goldenScenarios": [
63+
"call-resolution-stats",
64+
"unresolved-call-sites",
65+
"calls-method-ping-unresolved"
66+
],
67+
"knownGaps": [
68+
"method-call callee_symbol_id deferred until namespace/member resolve"
69+
]
70+
},
71+
{
72+
"id": "types.heritage",
73+
"description": "type_members, type_heritage, ancestor/descendant recipes",
74+
"fixtureFiles": [
75+
"src/types/hierarchy.ts",
76+
"src/types/heritage-qualified.ts",
77+
"src/types/homonym-mammal.ts"
78+
],
79+
"goldenScenarios": [
80+
"type-members-client-config",
81+
"type-ancestors-dog",
82+
"type-descendants-animal"
83+
]
84+
},
85+
{
86+
"id": "react.jsx",
87+
"description": "components, jsx_elements, jsx_attributes",
88+
"fixtureFiles": [
89+
"src/components/shop/ProductCard.tsx",
90+
"src/components/shop/ShopButton.tsx"
91+
],
92+
"goldenScenarios": [
93+
"components-no-false-positives",
94+
"find-jsx-usages",
95+
"jsx-attributes-product-card",
96+
"scopes-product-card"
97+
]
98+
},
99+
{
100+
"id": "css",
101+
"description": "css_variables, css_classes, css_keyframes, css imports",
102+
"fixtureFiles": ["src/theme.css", "src/styles/button.module.css"],
103+
"goldenScenarios": [
104+
"css-variables",
105+
"css-classes-module",
106+
"css-keyframes",
107+
"css-imports"
108+
]
109+
},
110+
{
111+
"id": "module.graph",
112+
"description": "barrels, cycles, dynamic imports, re_export_chains",
113+
"fixtureFiles": [
114+
"src/components/shop/index.ts",
115+
"src/lib/cache.ts",
116+
"src/lib/store.ts",
117+
"src/consumer.ts"
118+
],
119+
"goldenScenarios": [
120+
"find-barrel-files",
121+
"circular-imports",
122+
"dynamic-imports-prefetch",
123+
"re-export-chains-product-card",
124+
"barrel-chains"
125+
]
126+
},
127+
{
128+
"id": "behavioral",
129+
"description": "async_calls, try_catch, decorators, runtime_markers",
130+
"fixtureFiles": [
131+
"src/consumer.ts",
132+
"src/lib/cache.ts",
133+
"src/lib/complexity-fixture.ts",
134+
"src/api/decorated.ts",
135+
"src/env.ts"
136+
],
137+
"goldenScenarios": [
138+
"async-calls-prefetch",
139+
"find-swallowed-errors",
140+
"try-catch-rethrow-heuristics",
141+
"decorators-sealed",
142+
"runtime-markers-env",
143+
"find-leftover-console"
144+
]
145+
},
146+
{
147+
"id": "coverage.recipes",
148+
"description": "coverage ingest and killer / risk recipes",
149+
"fixtureFiles": ["coverage/coverage-final.json", "src/orphan.ts"],
150+
"goldenScenarios": [
151+
"coverage-rows-after-ingest",
152+
"untested-and-dead",
153+
"refactor-risk-ranking",
154+
"files-by-coverage"
155+
],
156+
"setup": ["ingest-coverage"]
157+
},
158+
{
159+
"id": "boundaries.suppressions",
160+
"description": "boundary_rules, suppressions, config-driven violations",
161+
"fixtureFiles": [
162+
".codemap/config.json",
163+
"src/components/shop/ApiBridge.tsx",
164+
"src/orphan.ts"
165+
],
166+
"goldenScenarios": [
167+
"boundary-rules-ui-no-api",
168+
"boundary-violations",
169+
"suppressions-orphan"
170+
]
171+
},
172+
{
173+
"id": "fts5",
174+
"description": "source_fts when fts5 enabled in config",
175+
"fixtureFiles": [".codemap/config.json"],
176+
"goldenScenarios": [
177+
"source-fts-row-count",
178+
"meta-fts5-enabled",
179+
"text-in-deprecated-functions"
180+
]
181+
},
182+
{
183+
"id": "tests.affected",
184+
"description": "test_suites and affected-tests recipe",
185+
"fixtureFiles": ["src/__tests__/smoke.test.ts"],
186+
"goldenScenarios": [
187+
"find-skipped-tests",
188+
"tests-by-file",
189+
"affected-tests-direct"
190+
]
191+
},
192+
{
193+
"id": "recipes.bundled",
194+
"description": "Every templates/recipes/*.sql id has a golden scenario",
195+
"fixtureFiles": [],
196+
"goldenScenarios": [],
197+
"enforcedBy": "scripts/query-golden-coverage-matrix.test.mjs"
198+
},
199+
{
200+
"id": "cli.mcp.http",
201+
"description": "Consumer surfaces (show, trace, audit, MCP, HTTP) — not golden-indexed",
202+
"fixtureFiles": [],
203+
"goldenScenarios": [],
204+
"unitTests": [
205+
"src/cli/cmd-cli-parity-e2e.test.ts",
206+
"src/application/mcp-server.test.ts",
207+
"src/application/tool-handlers.test.ts"
208+
],
209+
"knownGaps": ["No single spawned CLI matrix over full corpus yet"]
210+
}
211+
]
212+
}

‎fixtures/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Codemap in-repo test bench
2+
3+
**Single committed corpus** for regression-testing Codemap — no `CODEMAP_ROOT` pointing at external apps required for maintainers or CI.
4+
5+
| Piece | Path | Role |
6+
| ----------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
7+
| **Corpus** | [`minimal/`](./minimal/) | Source tree indexed by goldens, benchmark, agent-eval |
8+
| **Golden scenarios** | [`golden/scenarios.json`](./golden/scenarios.json) | 124+ SQL/recipe scenarios |
9+
| **Golden snapshots** | [`golden/minimal/`](./golden/minimal/) | Expected query JSON (committed) |
10+
| **Capability map** | [`CAPABILITIES.json`](./CAPABILITIES.json) | Machine-readable: capability → fixture files → scenarios |
11+
| **Agent eval** | [`agent-eval/`](../scripts/agent-eval/) + probes in `scripts/agent-eval/scenarios.json` | MCP-on vs traditional arms |
12+
| **Benchmark scenarios** | [`benchmark/scenarios.example.json`](./benchmark/scenarios.example.json) | Speed comparisons (optional override) |
13+
14+
## Commands (from repo root)
15+
16+
```bash
17+
bun run test:golden # index minimal + compare all scenarios
18+
bun run test:agent-eval # probe/live harness on same corpus
19+
bun run check # unit + golden + agent-eval (CI)
20+
bun scripts/query-golden.ts --update # refresh snapshots after intentional changes
21+
```
22+
23+
## Tier B / external trees
24+
25+
`bun run test:golden:external` remains for **consumers** validating Codemap against a private checkout (gitignored goldens). It is **not** part of the Codemap maintainer test bench.
26+
27+
Expansion plan: [docs/plans/in-repo-test-bench.md](../docs/plans/in-repo-test-bench.md).

0 commit comments

Comments
 (0)