Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/evidence-chains-recipes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@stainless-code/codemap": patch
---

Add `reason` and `evidence_json` columns on high-judgment recipe rows (`boundary-violations`, `deprecated-symbols`, `unimported-exports`) so agents can cite detection path before `apply` or manual edits.
2 changes: 2 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,8 @@ Three **mutually exclusive** CLI entry shapes; all converge on `applyDiffPayload

**Show / snippet wiring:** **`src/cli/show-snippet-args.ts`** (shared argv parser) + **`src/cli/show-snippet-render.ts`** (shared terminal/JSON error helpers) + **`src/cli/cmd-show.ts`** + **`src/cli/cmd-snippet.ts`** — sibling CLI verbs sharing the same parser shape (`<name>` or **`--query '<field:value …>'`** + **`--with-fts`** + `--kind` + `--in <path>` + `--json`; show adds **`--print-sql`**) and the pure engines **`src/application/show-engine.ts`** (exact lookup + envelope builders), **`src/application/search-query-parser.ts`** + **`src/application/search-engine.ts`** (field-qualified search → parameterized SQL on `symbols`, optional `source_fts` join), and **`src/application/show-search-mode.ts`** (shared parse/normalize + FTS resolution + **`executeShowLookup`** + **`formatShowSearchSqlForQuery`** for CLI/MCP/HTTP). Exact lookup: `findSymbolsByName({db, name, kind?, inPath?})`. Query lookup: `searchSymbols({db, parsed, withFts?})`. Snippet FS read: `readSymbolSource({match, projectRoot, indexedContentHash?})` + `getIndexedContentHash(db, filePath)`. **`buildShowResult`** + **`buildSnippetResult`** envelope builders — same engines the MCP show/snippet tools call. Both verbs return the same `{matches, disambiguation?, warning?}` envelope — single match → `{matches: [{...}]}`; multi-match adds `{n, by_kind, files, hint}`; optional **`warning`** when FTS was requested but `source_fts` is empty. Snippet matches add `source` / `stale` / `missing` fields (additive — no shape divergence). **`--in <path>`** and **`path:`** inside **`--query`** normalize through `toProjectRelative(projectRoot, p)` (from **`src/application/validate-engine.ts`**). Stale-file behavior on `snippet`: `hashContent` (from **`src/hash.ts`**) compares on-disk content against `files.content_hash`; mismatch sets `stale: true` but source IS still returned. MCP tools `show` and `snippet` register parallel to the CLI surface (see [§ MCP wiring](#cli-usage)).

**Evidence columns (high-judgment recipes):** Some bundled recipes add optional **`reason`** and **`evidence_json`** TEXT columns on each result row — factual detection path for agents, not pass/fail verdicts. Contract: [golden-queries.md § Evidence columns](./golden-queries.md#evidence-columns-high-judgment-recipes).

**Recipes wiring:** **`src/application/recipes-loader.ts`** (pure transport-agnostic loader) + **`src/application/query-recipes.ts`** (cache + public API — `getQueryRecipeSql` / `getQueryRecipeActions` / `getQueryRecipeParams` / `listQueryRecipeIds` / `listQueryRecipeCatalog` / `getQueryRecipeCatalogEntry`, shared by CLI + MCP). Recipes live as file pairs: **`<id>.sql`** + optional **`<id>.md`**. The loader reads `templates/recipes/` (bundled, ships in npm package next to `templates/agents/`) and `<state-dir>/recipes/` (project-local — default `.codemap/recipes/`; honors `--state-dir` / `CODEMAP_STATE_DIR`; root-only resolution per the registry plan, no walk-up). Project recipes win on id collision; entries that override a bundled id carry **`shadows: true`** in the catalog so agents reading `codemap://recipes` at session start see when a recipe behaves differently from the documented bundled version. Per-row **`actions`** templates and recipe **`params`** declarations live in YAML frontmatter on each `<id>.md` — uniform shape across bundled + project. Param types are `string | number | boolean`; CLI passes values via repeatable `--params key=value[,key=value]`, MCP / HTTP pass nested `params: {key: value}` to `query_recipe`. Validation runs before SQL binding; missing / unknown / malformed params return the same `{error}` envelope as query failures. Hand-rolled YAML parser is scoped to block-list `actions:` and `params:` only (no `js-yaml` dep). Load-time validation rejects empty SQL and DML / DDL keywords (`INSERT` / `UPDATE` / `DELETE` / `DROP` / `CREATE` / `ALTER` / `ATTACH` / `DETACH` / `REPLACE` / `TRUNCATE` / `VACUUM` / `PRAGMA`) with recipe-aware error messages — defence in depth alongside the runtime `PRAGMA query_only=1` backstop in `query-engine.ts` (PR #35). `<state-dir>/index.db` is gitignored; `<state-dir>/recipes/` is NOT (verified via `git check-ignore`) — recipes are git-tracked source code authored for human review.

**Tool / resource handlers (transport-agnostic):** **`src/application/tool-handlers.ts`** + **`src/application/resource-handlers.ts`** — pure functions that take the args object an MCP tool / resource URI accepts and return a discriminated **`ToolResult`** (`{ok: true, format: 'json'|'sarif'|'annotations'|'mermaid'|'diff'|'diff-json'|'codeclimate'|'badge', payload}` — badge arm also carries `badgeStyle`; `{ok: false, error}`) or a **`ResourcePayload`** (`{mimeType, text}`). MCP and HTTP both wrap the same handlers — MCP translates to `{content: [{type: "text", text}]}`, HTTP translates to `(status, body)` with the right `Content-Type`. Engine layer untouched; transport changes don't ripple into the SQL.
Expand Down
4 changes: 4 additions & 0 deletions docs/golden-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ Scenarios live in **`fixtures/golden/scenarios.json`** (Tier A) or optional **`s

**Prompts** in JSON are **intent labels**, not pasted chat logs — pair with queries whose literals come from **fixture-owned** data (see [fixtures/qa/prompts.external.template.md](../fixtures/qa/prompts.external.template.md) for optional chat QA).

### Evidence columns (high-judgment recipes)

Some bundled recipes add optional **`reason`** (TEXT) and **`evidence_json`** (TEXT, JSON array) columns on each row — factual detection path for agents, not engine verdicts. See [plans/evidence-chains-on-recipe-rows.md](./plans/evidence-chains-on-recipe-rows.md). Goldens assert these columns when the recipe ships evidence (`boundary-violations`, `deprecated-symbols`, `unimported-exports`).

---

## Status
Expand Down
94 changes: 94 additions & 0 deletions docs/plans/agent-enrichment-wave.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Agent enrichment wave — tracer workflow (plans 1–4)

> **Status:** in-flight · **Scope:** four P2 plans ranked by consumer/agent ROI
>
> **Goal:** Ship tracer bullets that cut agent round-trips, improve answer trust, and sharpen PR/CI deltas — all Moat-A (predicate columns, no verdict primitives).
>
> **Plans (execution order):** [evidence-chains](./evidence-chains-on-recipe-rows.md) → [graph-estimated-crap](./graph-estimated-crap.md) → [coverage-deletion-confidence](./coverage-deletion-confidence.md) → [audit-delta-attribution](./audit-delta-attribution.md)

---

## Shared conventions (locked)

| Convention | Applies to |
| ------------------------------------------------------------------------------------- | ---------- |
| **Moat A** — no `pass`/`fail` engine verdict; extra columns only | All four |
| **`reason` TEXT** — machine code + short clause where useful | #1, #3 |
| **`evidence_json` TEXT** — bounded JSON array (≤3 hops) | #1 |
| **`confidence` / `coverage_source` / `attribution`** — recipe-specific enums | #2, #3, #4 |
| **Golden update per slice** — `fixtures/golden/minimal/*.json` + `scenarios.json` | All |
| **`/harden-pr lite`** after each tracer commit; **`/harden-pr full`** before PR merge | All |

**Cross-plan synergy:** #1 `reason` on recipes complements #4 `attribution` on audit `added` rows (optional merge in evidence plan v2). #2 and #3 both touch coverage semantics — ship #2 before #3 so agents have CRAP tiers before deletion-confidence narrows rows.

---

## Plan 1 — Evidence chains (`evidence-chains-on-recipe-rows.md`)

| Slice | Deliverable | Verify |
| ----------------------------- | ------------------------------------------------------------------ | --------------------- |
| **1.0 contract** | `docs/golden-queries.md` § evidence columns; one architecture line | doc review |
| **1.1 `boundary-violations`** | `reason` + `evidence_json` in SQL; `.md` + golden | `bun run test:golden` |
| **1.2 `deprecated-symbols`** | caller hops in `evidence_json` | golden + matrix |
| **1.3 `unimported-exports`** | `re_export_chains` LEFT JOIN; `reason` variants | golden |
| **1.4 agent surface** | `templates/agent-content/rule/00-full.md` one-liner | consumer check |

**Open decisions (locked for v1):** E.2 `evidence_json` only (not typed columns); E.1 SQL-only (no query-engine post-processor).

---

## Plan 2 — Graph-estimated CRAP (`graph-estimated-crap.md`)

| Slice | Deliverable | Verify |
| ------------------------- | --------------------------------------------------------------- | ----------------- |
| **2.0 spike** | Reachability CTE on `fixtures/minimal` (script or ad-hoc query) | manual row counts |
| **2.1 recipe** | `high-crap-score.sql` + `.md`; `scenarios.json` | `test:golden` |
| **2.2 measured override** | golden with `ingest-coverage` setup | golden matrix |
| **2.3 cross-link** | `high-complexity-untested.md` points at CRAP when no ingest | doc |

**Grill before 2.1 if spike ambiguous:** Q1 type-only imports in walk (default: value edges only); Q2 recipe id `high-crap-score`.

---

## Plan 3 — Coverage deletion confidence (`coverage-deletion-confidence.md`)

| Slice | Deliverable | Verify |
| -------------------------- | -------------------------------------------------------------- | ------------- |
| **3.1 recipe fork** | `coverage-confirmed-dead.sql` + `.md` from `untested-and-dead` | query CLI |
| **3.2 golden no-ingest** | `confidence: medium` policy (per D.4) | `test:golden` |
| **3.3 golden with ingest** | fixture coverage → `confidence: high` | `test:golden` |
| **3.4 classifier** | intent keywords if needed | optional |

**Grill before 3.1:** Q3 without ingest — `medium` rows vs empty + stderr (plan D.4 leans medium rows).

---

## Plan 4 — Audit delta attribution (`audit-delta-attribution.md`)

| Slice | Deliverable | Verify |
| -------------------------------- | --------------------------------------- | ---------------------- |
| **4.1 `findingKey()`** | pure helper + unit tests | `audit-engine.test.ts` |
| **4.2 `deprecated` delta** | `attribution` on `added[]` for `--base` | branch fixture test |
| **4.3 `files` + `dependencies`** | generalize key sets from base cache | tests |
| **4.4 transport parity** | MCP/HTTP envelope match CLI | handler test |
| **4.5 docs** | `architecture.md` envelope § | doc |

---

## PR cadence

| PR | Contents | Changeset |
| -------------------------- | ------------------------------------ | --------- |
| **#A Evidence wave 1** | Slices 1.0–1.1 (boundary-violations) | patch |
| **#B Evidence wave 2** | Slices 1.2–1.4 | patch |
| **#C CRAP recipe** | Plan 2 complete | patch |
| **#D Deletion confidence** | Plan 3 complete | patch |
| **#E Audit attribution** | Plan 4 complete | patch |

Each PR: `harden-pr full` → merge. Do not batch plans 1–4 into one PR.

---

## Current slice

**Active:** Plan 1 shipped in [**PR #174**](https://github.com/stainless-code/codemap/pull/174) (awaiting merge) — next: Plan 2 spike **2.0** (`graph-estimated-crap.md`).
22 changes: 11 additions & 11 deletions docs/plans/evidence-chains-on-recipe-rows.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Evidence is **in-SQL**, not a post-processor — same Moat-A path as the recipe.

### Tracer bullet (slice 1)

`boundary-violations`: add `reason` constant + `evidence_json` with rule tuple; one golden query. Ship before touching `unimported-exports` re-export subquery.
`boundary-violations`: add `reason` constant + `evidence_json` with rule tuple; one golden query. Ship before touching `unimported-exports` re-export subquery. **Orchestration:** [agent-enrichment-wave.md](./agent-enrichment-wave.md) § Plan 1 slice 1.1 — shipped `edeee68`.

### Out of scope (v1)

Expand Down Expand Up @@ -98,21 +98,21 @@ Ship one recipe per wave; verify before moving to the next.

## Acceptance

- [ ] `codemap query --recipe unimported-exports --json` rows include `reason`; re-export false-positive class includes non-empty `evidence_json` when chain exists
- [ ] `boundary-violations` rows include stable `reason: boundary_deny_match`
- [ ] `deprecated-symbols` rows with callers include `evidence_json` caller hops
- [ ] Golden queries updated; no new CLI verb
- [ ] SARIF / annotations unchanged (extra columns ignored by formatters unless future mapping added)
- [x] `codemap query --recipe unimported-exports --json` rows include `reason`; re-export false-positive class includes non-empty `evidence_json` when chain exists
- [x] `boundary-violations` rows include stable `reason: boundary_deny_match`
- [x] `deprecated-symbols` rows with callers include `evidence_json` caller hops
- [x] Golden queries updated; no new CLI verb
- [x] SARIF / annotations unchanged (extra columns ignored by formatters unless future mapping added)

---

## Open decisions (impl PR)

| # | Question |
| --- | ------------------------------------------------------------------------------------------------- |
| Q1 | Single `evidence_json` vs separate typed columns (`reexport_hops`, `caller_count`)? |
| Q2 | Post-query enrichment in `query-engine` for recipes that opt in via frontmatter `evidence: true`? |
| Q3 | Include `binding_kind` from `bindings` for rename-preview synergy in v1 or v2? |
| # | Question | Lock (wave 2026-06) |
| --- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| Q1 | Single `evidence_json` vs separate typed columns (`reexport_hops`, `caller_count`)? | **`evidence_json` only** (E.2) — one JSON contract per recipe row |
| Q2 | Post-query enrichment in `query-engine` for recipes that opt in via frontmatter `evidence: true`? | **SQL-only** (E.1) — no post-processor in v1 |
| Q3 | Include `binding_kind` from `bindings` for rename-preview synergy in v1 or v2? | **v2** |

---

Expand Down
4 changes: 3 additions & 1 deletion fixtures/golden/minimal/boundary-violations.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
"to_path": "src/api/client.ts",
"rule_name": "ui-no-api",
"rule_from_glob": "src/components/**",
"rule_to_glob": "src/api/**"
"rule_to_glob": "src/api/**",
"reason": "boundary_deny_match",
"evidence_json": "[{\"rule_name\":\"ui-no-api\",\"from_glob\":\"src/components/**\",\"to_glob\":\"src/api/**\"}]"
}
]
4 changes: 2 additions & 2 deletions fixtures/golden/minimal/call-resolution-stats.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[
{
"total_calls": 47,
"resolved_calls": 26,
"total_calls": 48,
"resolved_calls": 27,
"method_calls_deferred": 20,
"unresolved_queue": 1,
"residual_meta": "1"
Expand Down
7 changes: 7 additions & 0 deletions fixtures/golden/minimal/components-touching-deprecated.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
[
{
"component": "ProductCard",
"component_file": "src/components/shop/ProductCard.tsx",
"deprecated_symbol": "now",
"deprecated_file": "src/utils/date.ts",
"via": "call"
},
{
"component": "ShopButton",
"component_file": "src/components/shop/ShopButton.tsx",
Expand Down
11 changes: 9 additions & 2 deletions fixtures/golden/minimal/coverage-rows-after-ingest.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,15 @@
{
"file_path": "src/components/shop/ProductCard.tsx",
"name": "ProductCard",
"hit_statements": 3,
"total_statements": 3,
"hit_statements": 2,
"total_statements": 2,
"coverage_pct": 100
},
{
"file_path": "src/components/shop/ProductCard.tsx",
"name": "spread",
"hit_statements": 1,
"total_statements": 1,
"coverage_pct": 100
},
{
Expand Down
12 changes: 9 additions & 3 deletions fixtures/golden/minimal/deprecated-symbols.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,28 @@
"file_path": "src/api/client.ts",
"line_start": 46,
"signature": "legacyClient()",
"doc_comment": "@deprecated Use `createClient({ baseUrl })` directly. Kept as a fixture for\n`deprecated-symbols` + `--format sarif` / `--format annotations` recipes."
"doc_comment": "@deprecated Use `createClient({ baseUrl })` directly. Kept as a fixture for\n`deprecated-symbols` + `--format sarif` / `--format annotations` recipes.",
"reason": "no_callers",
"evidence_json": "[]"
},
{
"name": "now",
"kind": "function",
"file_path": "src/utils/date.ts",
"line_start": 5,
"signature": "now(): number",
"doc_comment": "@deprecated Use `Date.now()` directly. Kept as a fixture for the\n`deprecated-symbols` recipe golden test."
"doc_comment": "@deprecated Use `Date.now()` directly. Kept as a fixture for the\n`deprecated-symbols` recipe golden test.",
"reason": "has_callers",
"evidence_json": "[{\"kind\":\"caller\",\"name\":\"ProductCard\",\"file_path\":\"src/components/shop/ProductCard.tsx\",\"line_start\":12},{\"kind\":\"caller\",\"name\":\"ShopButton\",\"file_path\":\"src/components/shop/ShopButton.tsx\",\"line_start\":10},{\"kind\":\"caller\",\"name\":\"run\",\"file_path\":\"src/consumer.ts\",\"line_start\":27},{\"truncated\":true}]"
},
{
"name": "epochMs",
"kind": "function",
"file_path": "src/utils/format.ts",
"line_start": 5,
"signature": "epochMs(): number",
"doc_comment": "@deprecated Drift detector for the SARIF / GH-annotations golden output.\nPair with `now()` in `./date.ts` to give recipes >1 row to render."
"doc_comment": "@deprecated Drift detector for the SARIF / GH-annotations golden output.\nPair with `now()` in `./date.ts` to give recipes >1 row to render.",
"reason": "has_callers",
"evidence_json": "[{\"kind\":\"caller\",\"name\":\"run\",\"file_path\":\"src/consumer.ts\",\"line_start\":30}]"
}
]
4 changes: 2 additions & 2 deletions fixtures/golden/minimal/files-hashes.json
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,9 @@
},
{
"path": "src/components/shop/ProductCard.tsx",
"content_hash": "27b25a905655574a3002adc140b36e91b95f8f1a5db0758ba172115595134122",
"content_hash": "130e72381a5b03071f22f161e178d2800c48124d1bbc33359520218ad48e3a1e",
"language": "tsx",
"line_count": 23
"line_count": 24
},
{
"path": "src/components/shop/ShopButton.default.ts",
Expand Down
4 changes: 2 additions & 2 deletions fixtures/golden/minimal/files-largest.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@
},
{
"path": "src/components/shop/ProductCard.tsx",
"line_count": 23,
"size": 654,
"line_count": 24,
"size": 748,
"language": "tsx"
},
{
Expand Down
2 changes: 1 addition & 1 deletion fixtures/golden/minimal/find-export-sites.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"is_re_export": 0,
"re_export_source": null,
"line_start": 10,
"line_end": 22,
"line_end": 23,
"column_start": 16,
"column_end": 27
},
Expand Down
4 changes: 2 additions & 2 deletions fixtures/golden/minimal/find-jsx-usages.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
{
"file_path": "src/components/shop/ProductCard.tsx",
"component_name": "article",
"line_start": 15,
"line_end": 19,
"line_start": 16,
"line_end": 20,
"is_self_closing": 0,
"is_fragment": 0,
"is_lowercase": 1,
Expand Down
2 changes: 1 addition & 1 deletion fixtures/golden/minimal/find-symbol-definitions.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"name": "ProductCard",
"kind": "function",
"line_start": 10,
"line_end": 22,
"line_end": 23,
"name_column_start": 16,
"name_column_end": 27,
"parent_name": null,
Expand Down
2 changes: 1 addition & 1 deletion fixtures/golden/minimal/index-summary.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[
{
"files": 42,
"symbols": 105,
"symbols": 106,
"imports": 25,
"components": 5,
"dependencies": 23
Expand Down
Loading
Loading