Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/churn-complexity-hotspots.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@stainless-code/codemap": minor
---

Add churn × complexity hotspot ranking: `file_churn` refreshed on every index from git history, with `codemap ingest-churn` and config `churn.file` for non-git repos. New `churn-complexity-hotspots` recipe ranks files or symbols (`by_symbol`) by change frequency × complexity with normalized 0–100 scores and `churn_trend`. Outcome alias `hotspots` still maps to fan-in.
19 changes: 18 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ Three **mutually exclusive** CLI entry shapes; all converge on `applyDiffPayload

**`src/application/session-lifecycle.ts`** — transport-specific start/stop rules for long-running `mcp` / `serve` processes (one-shot CLI unchanged). **`createStdioDisconnectMonitor`** (MCP only) exits the process when the agent host is actually gone: stdin EOF, stdout `EPIPE`, boot parent PID no longer alive (2s poll), or SIGINT/SIGTERM. The MCP SDK's stdio `transport.onclose` alone is insufficient — it fires only after an explicit `transport.close()`, not when the parent crashes without tearing down the pipe. **`createManagedWatchSession`** refcount-gates chokidar: MCP acquires one client before `connect` and **`forceStop`** drains the watcher on disconnect; HTTP acquires per authenticated request (after auth; **`GET /health`** excluded) and **`releaseClient`** stops the watcher when the count hits zero. **No MCP idle timeout:** `codemap mcp` does **not** exit after N minutes without tool calls while the stdio pipe stays open. IDE hosts spawn MCP once per session and do not reliably respawn it mid-conversation — an idle shutdown would break long pauses (human think time, reading, multi-step plans) with no recovery path. Orphan cleanup is handled by **disconnect detection**, not inactivity timers. **HTTP watch release grace (`HTTP_WATCH_RELEASE_GRACE_MS` = 5000):** distinct from idle timeout — only stops chokidar between stateless requests so the watcher is not started/stopped on every POST; the HTTP listener keeps running. **`GET /health`** liveness probes do not acquire a watch client (probes must not keep chokidar hot). Future **`MCP shared daemon per project`** could revisit opt-in idle policies with explicit client reconnect; not planned for stdio MCP today.

**Performance wiring:** **`--performance`** plumbs through **`RunIndexOptions.performance`** → **`indexFiles({ performance, collectMs })`**. `parse-worker-core.ts` records per-file **`parseMs`** on each `ParsedFile`; main thread times the eight phases (`collect`, `parse`, `insert`, `index_create`, `bindings`, `module_cycles`, `re_export_chains`, `heritage`) and assembles **`IndexPerformanceReport`** under `IndexRunStats.performance`. Note: `total_ms` is `indexFiles` wall-clock (parse + insert + DDL + bindings + cycles + re_exports + heritage), **not** end-to-end run wall — `collect_ms` happens before `indexFiles` and is reported separately. Env var **`CODEMAP_PERFORMANCE_JSON=<path>`** dumps the report as JSON post-run (consumed by [`bun run check:perf-baseline`](./benchmark.md#perf-baseline-regression-guardrail) for local + weekly scheduled drift checks — not a PR merge gate).
**Performance wiring:** **`--performance`** plumbs through **`RunIndexOptions.performance`** → **`indexFiles({ performance, collectMs })`**. `parse-worker-core.ts` records per-file **`parseMs`** on each `ParsedFile`; main thread times the eight phases (`collect`, `parse`, `insert`, `index_create`, `bindings`, `module_cycles`, `re_export_chains`, `heritage`) and assembles **`IndexPerformanceReport`** under `IndexRunStats.performance`. Post-index **`refreshFileChurn`** records **`churn_ms`** separately (patched into the performance JSON when `CODEMAP_PERFORMANCE_JSON` is set). Note: `total_ms` is `indexFiles` wall-clock (parse + insert + DDL + bindings + cycles + re_exports + heritage), **not** end-to-end run wall — `collect_ms` and `churn_ms` happen outside `indexFiles` and are reported separately. Env var **`CODEMAP_PERFORMANCE_JSON=<path>`** dumps the report as JSON post-run (consumed by [`bun run check:perf-baseline`](./benchmark.md#perf-baseline-regression-guardrail) for local + weekly scheduled drift checks — not a PR merge gate).

**Agent templates:** `codemap agents init` writes thin pointer files (~18-line SKILL + ~25-line rule) to consumer disk; full content is served live by `codemap skill` / `codemap rule` (CLI) and `codemap://skill` / `codemap://rule` (MCP / HTTP) from `templates/agent-content/<kind>/*.md`. Section files concatenate in lexical order; `*.gen.md` sections dispatch to renderers in `application/agent-content.ts` so recipe catalog + schema DDL auto-register. Pointer-version stamp (`<!-- codemap-pointer-version: N -->`) + once-per-process stderr nag (`maybeWarnStalePointers`) flag stale consumer templates; cure is `codemap agents init --force`. Full matrix: [agents.md](./agents.md).

Expand Down Expand Up @@ -496,6 +496,23 @@ One row per leaf parameter binding, ordered by `position`. Pattern params (`func
| column_start | INTEGER | 0-based column of the binding token |
| column_end | INTEGER | One-past-last column |

### `file_churn` — Git churn metrics per indexed file (`STRICT`)

One row per indexed file with git history in scope. Populated on **every index pass** by `refreshFileChurn` → `ingestFileChurnFromGit` (`git log --numstat` scoped to the project root pathspec). Tunable via config `churn.halfLifeDays` (default 90) and optional `churn.since` / CLI `--churn-since <ref>`. Non-git repos skip ingest (empty table; recipe returns no rows). `churn_trend` is `accelerating` \| `stable` \| `cooling` when enough history exists, else NULL.

| Column | Type | Description |
| ---------------- | ------- | -------------------------------------------------------------------------- |
| file_path | TEXT PK | FK → `files(path)` CASCADE |
| commit_count | INTEGER | Distinct commits touching the file in scope |
| weighted_commits | REAL | Recency-weighted commit count (default 90-day half-life exponential decay) |
| lines_added | INTEGER | Sum of added lines from numstat |
| lines_removed | INTEGER | Sum of removed lines from numstat |
| last_commit_at | TEXT | ISO timestamp of most recent commit touching the file |
| churn_trend | TEXT | `"accelerating"` \| `"stable"` \| `"cooling"` — nullable in v1 |
| computed_at | TEXT | ISO timestamp when ingest last ran |

Powers **`churn-complexity-hotspots`** recipe (`hotspot_score`, `hotspot_score_normalized`; file or symbol grain via `by_symbol`). Non-git: **`codemap ingest-churn`** or config **`churn.file`**. Distinct from outcome alias **`hotspots`** → `fan-in`.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

### `file_metrics` — Per-file aggregate metrics (`STRICT`)

One row per indexed TS/JS file. Line classification is regex-light (blank if `/^\s*$/`; comment if line starts with `//`, `/*`, `*`, `*/`).
Expand Down
4 changes: 2 additions & 2 deletions docs/benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,9 @@ Independent of the consumer-facing scenarios above, the repo carries a **per-pha

### Mechanism

1. `bun src/index.ts --full --performance` populates [`IndexPerformanceReport`](../src/application/types.ts) with `collect_ms` / `parse_ms` / `insert_ms` / `index_create_ms` / `bindings_ms` / `module_cycles_ms` / `re_export_chains_ms` / `heritage_ms` / `total_ms`.
1. `bun src/index.ts --full --performance` populates [`IndexPerformanceReport`](../src/application/types.ts) with `collect_ms` / `parse_ms` / `insert_ms` / `index_create_ms` / `bindings_ms` / `module_cycles_ms` / `re_export_chains_ms` / `heritage_ms` / `total_ms`, plus post-index **`churn_ms`** (git churn ingest; patched after `indexFiles` completes).
2. Setting `CODEMAP_PERFORMANCE_JSON=<path>` dumps that report as JSON to `<path>` after the run (no CLI flag added; env-var only).
3. [`scripts/check-perf-baseline.ts`](../scripts/check-perf-baseline.ts) (alias `bun run check:perf-baseline`) runs the indexer 3× on this repo, takes per-phase **medians**, and compares **`collect_ms`**, **`parse_ms`**, **`insert_ms`**, **`index_create_ms`**, **`bindings_ms`**, and **`total_ms`** to `fixtures/benchmark/perf-baseline.json`. Other `IndexPerformanceReport` fields (`module_cycles_ms`, `re_export_chains_ms`, `heritage_ms`, …) appear in `--performance` JSON only — not baseline-gated.
3. [`scripts/check-perf-baseline.ts`](../scripts/check-perf-baseline.ts) (alias `bun run check:perf-baseline`) runs the indexer 3× on this repo, takes per-phase **medians**, and compares **`collect_ms`**, **`parse_ms`**, **`insert_ms`**, **`index_create_ms`**, **`bindings_ms`**, **`churn_ms`**, and **`total_ms`** to `fixtures/benchmark/perf-baseline.json`. Other `IndexPerformanceReport` fields (`module_cycles_ms`, `re_export_chains_ms`, `heritage_ms`, …) appear in `--performance` JSON only — not baseline-gated.
4. **Local / scheduled only** — run before perf-sensitive PRs; [`.github/workflows/perf-baseline.yml`](../.github/workflows/perf-baseline.yml) fires weekly + `workflow_dispatch` for drift visibility. **Not** on the PR CI path (6 min × 3 runs + bimodal GHA runners → flaky merge gate).

### Why this is separate from `src/benchmark.ts`
Expand Down
6 changes: 5 additions & 1 deletion docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,10 @@ Number of edges _into_ a file in the `dependencies` table — `COUNT(*) FROM dep

Number of edges _out of_ a file — `COUNT(*) FROM dependencies WHERE from_path = ?`. Surfaces as the `fan-out` recipe.

### `file_churn` (table)

Per-file git churn metrics refreshed on **every index pass** (incremental scoped recompute; idle HEAD cache). Git via `ingestFileChurnFromGit`; non-git via **`codemap ingest-churn`** or config **`churn.file`**. Config: `churn.halfLifeDays`, `churn.since` / `--churn-since`. Column **`churn_trend`**: `accelerating` \| `stable` \| `cooling` when enough history exists. Powers **`churn-complexity-hotspots`** (file or symbol grain, normalized score) — distinct from outcome alias **`hotspots`** → `fan-in`.

### `files` (table)

Header row for every indexed file. `path` is the primary key; all other tables FK to it with `ON DELETE CASCADE`. Flags: `is_barrel` (100% re-exports, no local value symbols) and `has_side_effects` (module-level call/assignment seen at parse time). See `FileRow`.
Expand Down Expand Up @@ -410,7 +414,7 @@ Key-value metadata table. Holds `schema_version`, `last_indexed_commit`, `indexe

### outcome aliases (`dead-code` / `deprecated` / `boundaries` / `hotspots` / `coverage-gaps`)

Top-level CLI verbs that thin-wrap `query --recipe <id>`: `dead-code` → `untested-and-dead`, `deprecated` → `deprecated-symbols`, `boundaries` → `boundary-violations`, `hotspots` → `fan-in`, `coverage-gaps` → `worst-covered-exports`. Every `query` flag passes through (`--json`, `--format`, `--ci`, `--summary`, `--changed-since`, `--group-by`, `--params`, `--save-baseline`, `--baseline`). Mapping lives in `src/cli/aliases.ts` (`OUTCOME_ALIASES`). Capped at 5 to avoid alias-sprawl — promote a sixth only when the recipe becomes a headline outcome. Moat-A clean: the alias is a one-line rewrite, not a new primitive; the recipe IS the SQL. **Write alias (distinct):** `codemap rename` thin-wraps `apply rename-preview` (not `query --recipe`) — mapping in `src/cli/rename-alias.ts`; same Moat-A rule (no new write semantics).
Top-level CLI verbs that thin-wrap `query --recipe <id>`: `dead-code` → `untested-and-dead`, `deprecated` → `deprecated-symbols`, `boundaries` → `boundary-violations`, `hotspots` → `fan-in`, `coverage-gaps` → `worst-covered-exports`. For **change-frequency × complexity** refactor targets use recipe **`churn-complexity-hotspots`** (not the `hotspots` alias). Every `query` flag passes through (`--json`, `--format`, `--ci`, `--summary`, `--changed-since`, `--group-by`, `--params`, `--save-baseline`, `--baseline`). Mapping lives in `src/cli/aliases.ts` (`OUTCOME_ALIASES`). Capped at 5 to avoid alias-sprawl — promote a sixth only when the recipe becomes a headline outcome. Moat-A clean: the alias is a one-line rewrite, not a new primitive; the recipe IS the SQL. **Write alias (distinct):** `codemap rename` thin-wraps `apply rename-preview` (not `query --recipe`) — mapping in `src/cli/rename-alias.ts`; same Moat-A rule (no new write semantics).

### oxc-parser

Expand Down
6 changes: 5 additions & 1 deletion docs/golden-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ We **do not** commit another product’s source tree, paths, business strings, o

## Scenario shape (implemented)

Scenarios live in **`fixtures/golden/scenarios.json`** (Tier A) or optional **`scenarios.external.json`** / **example** (Tier B). The file may be a **bare array** of scenarios (legacy) or an object `{ "setup": [...], "scenarios": [...] }`. Optional top-level **`setup`** runs once after index, before scenarios — today **`ingest-coverage`** and **`clear-coverage`** (see [run-setup.ts](../scripts/query-golden/run-setup.ts)); missing coverage files are skipped with a warning. Per-scenario **`preSetup`** runs after global setup (global setup restores after the scenario when `preSetup` mutates the index). Each scenario has **`id`**, **`sql` or `recipe`**, optional **`match`** (`exact`, `minRows`, `everyRowContains`, `everyRowFieldEquals`), optional **`budgetMs`**. Goldens: **`fixtures/golden/minimal/*.json`** etc. Refresh: **`bun scripts/query-golden.ts --update`**.
Scenarios live in **`fixtures/golden/scenarios.json`** (Tier A) or optional **`scenarios.external.json`** / **example** (Tier B). The file may be a **bare array** of scenarios (legacy) or an object `{ "setup": [...], "scenarios": [...] }`. Optional top-level **`setup`** runs once after index, before scenarios — today **`ingest-coverage`**, **`clear-coverage`**, and **`seed-file-churn`** (see [run-setup.ts](../scripts/query-golden/run-setup.ts)); missing coverage files are skipped with a warning. Per-scenario **`preSetup`** runs after global setup (global setup restores after the scenario when `preSetup` mutates the index). Each scenario has **`id`**, **`sql` or `recipe`**, optional **`match`** (`exact`, `minRows`, `everyRowContains`, `everyRowFieldEquals`), optional **`budgetMs`**. Goldens: **`fixtures/golden/minimal/*.json`** etc. Refresh: **`bun scripts/query-golden.ts --update`**.

**Prompts** in JSON are **intent labels**, not pasted chat logs — pair with queries whose literals come from **fixture-owned** data (see [fixtures/qa/prompts.external.template.md](../fixtures/qa/prompts.external.template.md) for optional chat QA).

Expand All @@ -78,6 +78,10 @@ Some bundled recipes add optional **`reason`** (TEXT) and **`evidence_json`** (T

`coverage-confirmed-dead` adds **`confidence`** (`high` \| `medium`) on each row — **`high`** when static dead and ingested `coverage_pct = 0`; **`medium`** when static dead but the symbol has no ingested coverage row. Also **`reason`**, **`caller_count`**. Goldens: `coverage-confirmed-dead` (post-ingest mix) and `coverage-confirmed-dead-no-ingest` (`preSetup: clear-coverage`, `everyRowFieldEquals` on `confidence: medium`).

### Churn / hotspot columns (`churn-complexity-hotspots` recipe)

`churn-complexity-hotspots` ranks indexed files or symbols by git churn × cyclomatic complexity. File grain (default): **`file_path`**, **`max_complexity`**, **`weighted_commits`**, **`commit_count`**, **`churn_trend`**, **`hotspot_score`**, **`hotspot_score_normalized`**. Symbol grain (`by_symbol=true`): per-symbol **`name`**, **`kind`**, **`line_start`**, **`cyclomatic_complexity`** plus file churn fields. Goldens: `churn-complexity-hotspots`, `churn-complexity-hotspots-by-symbol` (fixture churn seeded via setup step **`seed-file-churn`**).

### Duplication columns (`duplicates` recipe)

`duplicates` returns one row per function-shaped symbol in a **`body_hash`** collision group: **`name`**, **`kind`**, **`file_path`**, **`line_start`**, **`line_end`**, **`body_hash`**, **`body_line_count`**, **`duplicate_count`** (in-scope group size after `path_prefix` / `min_body_lines`). Substrate column **`symbols.body_hash`** is populated at index for function-shaped symbols (`function`, `method`, `getter`, `setter`) when `body_line_count >= 2`. Goldens: `duplicates` (includes `src/bench/duplicate-body-{a,b}.ts` pair). False positives possible when unrelated functions share control-flow skeleton or sync vs async/generator bodies match — triage with `snippet`. Recipe caps at **50 rows** (no truncation marker).
Expand Down
Loading
Loading