diff --git a/.agents/rules/codemap.mdc b/.agents/rules/codemap.mdc index ba122a7c..0a62c771 100644 --- a/.agents/rules/codemap.mdc +++ b/.agents/rules/codemap.mdc @@ -16,6 +16,9 @@ A local database (default **`.codemap.db`**) indexes structure: symbols, imports | ------- | ----------------- | ----- | | **Default** — from this clone | `bun src/index.ts` | `bun src/index.ts query ""` | | Same entry | `bun run dev` | (same as first row) | +| JSON output | — | `bun src/index.ts query --json ""` | +| Recipe | — | `bun src/index.ts query --recipe fan-out` (see **`bun src/index.ts query --help`**) | +| Recipe catalog / SQL | — | `bun src/index.ts query --recipes-json` · `bun src/index.ts query --print-sql fan-out` | After **`bun run build`**, **`node dist/index.mjs`** matches the published **`codemap`** binary (same flags). **`bun link`** / global **`codemap`** also work when testing the packaged CLI. @@ -66,6 +69,10 @@ If the question looks like any of these → use the index: bun src/index.ts query "" ``` +**Row count:** The CLI does **not** impose a maximum number of rows. Add **`LIMIT`** (and **`ORDER BY`**) in SQL when you need a bounded list. For automation and multi-row answers, prefer **`bun src/index.ts query --json`** — stdout is a JSON array; on failure, stdout is **`{"error":""}`** and the process exits **1**. + +**Verbatim answers:** When the user asks for lists, counts, or enumerated structural data from the index, **paste or summarize from the query output without inventing rows** — do not substitute a prose “summary” that omits rows the user asked to see. Prefer **`--json`** so the full result set is unambiguous. + ## Quick reference queries | I need to... | Query | diff --git a/.agents/skills/codemap/SKILL.md b/.agents/skills/codemap/SKILL.md index 64d38e13..fe37883c 100644 --- a/.agents/skills/codemap/SKILL.md +++ b/.agents/skills/codemap/SKILL.md @@ -16,6 +16,44 @@ bun src/index.ts query "" After **`bun run build`**, **`node dist/index.mjs query …`** or a linked **`codemap`** binary matches the published CLI. Use **`--root`** / **`CODEMAP_ROOT`** to index another tree. +## Query output and agents + +- **`bun src/index.ts query --json`** prints a **JSON array** of row objects to stdout. On SQL error, stdout is **`{"error":""}`** and the process exits **1**. +- The CLI **does not cap** how many rows SQLite returns — add **`LIMIT`** and **`ORDER BY`** in SQL when you need a bounded list. +- When answering with structural facts from the index (lists of paths, symbols, dependency edges), **ground the answer in the query rows** — do not invent or silently drop rows. Prefer **`--json`** for large or multi-column results. + +## Agent-friendly SQL recipes + +Replace placeholders (`'...'`) with your module path, file glob, or symbol name. + +**CLI shortcuts:** **`bun src/index.ts query --recipe `** runs bundled SQL (optional **`--json`**). **`bun src/index.ts query --recipes-json`** prints every bundled recipe (**`id`**, **`description`**, **`sql`**) as JSON (no index / DB required). **`bun src/index.ts query --print-sql `** prints one recipe’s SQL only. Ids include **`fan-out`**, **`fan-out-sample`** (**`GROUP_CONCAT`** samples), **`fan-out-sample-json`** (same, but **`json_group_array`** — needs SQLite JSON1), **`fan-in`**, **`index-summary`**, **`files-largest`**, **`components-by-hooks`**, **`markers-by-kind`** — see **`bun src/index.ts query --help`**. The fan-out rows match the SQL below; others align with “Conditional aggregation”, “Codebase statistics”, and component sections later in this skill. + +**Top files by dependency fan-out:** + +```sql +SELECT from_path, COUNT(*) AS deps +FROM dependencies +GROUP BY from_path +ORDER BY deps DESC +LIMIT 10; +``` + +**Same ranking, plus up to five sample targets per file** (uses a correlated subquery; adjust **`LIMIT 5`** as needed): + +```sql +SELECT d.from_path, + COUNT(*) AS deps, + (SELECT GROUP_CONCAT(to_path, ' | ') + FROM (SELECT to_path FROM dependencies d2 WHERE d2.from_path = d.from_path LIMIT 5)) + AS sample_targets +FROM dependencies d +GROUP BY d.from_path +ORDER BY deps DESC +LIMIT 10; +``` + +**JSON array samples (JSON1):** replace **`GROUP_CONCAT`** with **`json_group_array(to_path)`** in that subquery if your SQLite build has JSON1 — or use **`bun src/index.ts query --recipe fan-out-sample-json`**. + ## Schema ### `files` — Every indexed file @@ -196,6 +234,8 @@ SELECT name, file_path, hooks_used FROM components WHERE hooks_used LIKE '%useTheme%'; -- Components with most hooks (complexity indicator) +-- `json_array_length` requires SQLite JSON1. For a portable ranking, use +-- `bun src/index.ts query --recipe components-by-hooks` (comma-based count on the stored JSON array). SELECT name, file_path, json_array_length(hooks_used) as hook_count FROM components ORDER BY hook_count DESC LIMIT 15; diff --git a/.changeset/codemap-query-and-golden.md b/.changeset/codemap-query-and-golden.md new file mode 100644 index 00000000..729f785e --- /dev/null +++ b/.changeset/codemap-query-and-golden.md @@ -0,0 +1,34 @@ +--- +"@stainless-code/codemap": patch +--- + +**Query CLI** + +- **`codemap query --json`**: print a JSON array of result rows to stdout (and **`{"error":"…"}`** on SQL errors) for agents and automation. Document that the query subcommand does **not** cap rows — use SQL **`LIMIT`** for bounded results. Update bundled agent rule and skill with **`--json`** preference, verbatim structural answers, and generic SQL recipes (fan-out + sample targets). + +- **`codemap query --recipe `** for bundled read-only SQL so agents can run common structural queries without embedding SQL on the command line. **`--json`** works with recipes the same way as ad-hoc SQL. Bundled ids include dependency **`fan-out`** / **`fan-out-sample`** / **`fan-out-sample-json`** (JSON1 **`json_group_array`**) / **`fan-in`**, index **`index-summary`**, **`files-largest`**, React **`components-by-hooks`** (comma-based hook count, no JSON1), and **`markers-by-kind`**. The benchmark suite uses the **`fan-out`** recipe SQL for an indexed-path scenario; docs clarify that recipes add no extra query cost vs pasting the same SQL. + +- **Recipe discovery (no index / DB):** **`codemap query --recipes-json`** prints all bundled recipes (**`id`**, **`description`**, **`sql`**) as JSON. **`codemap query --print-sql `** prints one recipe’s SQL. **`listQueryRecipeCatalog()`** in **`src/cli/query-recipes.ts`** is the single derived view of **`QUERY_RECIPES`** for the JSON output. + +**Golden tests** + +- **`bun run test:golden`**: index **`fixtures/minimal`**, run scenarios from **`fixtures/golden/scenarios.json`**, and compare query JSON to **`fixtures/golden/minimal/`**. Use **`bun scripts/query-golden.ts --update`** after intentional fixture or schema changes. Documented in **benchmark.md** and **CONTRIBUTING**. + +**Query robustness** + +- With **`--json`**, **`{"error":"…"}`** is printed for invalid SQL, database open failures, and **`codemap query`** bootstrap failures (config / resolver setup), not only bad SQL. The CLI sets **`process.exitCode`** instead of **`process.exit`** so piped stdout is not cut off mid-stream. + +**Benchmark & `CODEMAP_BENCHMARK_CONFIG`** + +- Each **`indexedSql`** in custom scenario JSON is validated as a single read-only **`SELECT`** (or **`WITH` … `SELECT`**) — DDL/DML and **`RETURNING`** are rejected before execution. +- Config file paths are resolved from **`process.cwd()`** (see **benchmark.md**). **`traditional.regex`** strings are developer-controlled (local JSON); **`files`** mode compiles the regex once per scenario. +- Overlapping **globs** in the traditional path are **deduplicated** so **Files read** / **Bytes read** count each path once. +- The default **components in `shop/`** scenario uses a **`LIKE`** filter aligned with the traditional globs under **`components/shop/`** (**\*.tsx** and **\*.jsx**, matching **`components`** rows from the parser) and avoids unrelated paths such as **`workshop`**. + +**Recipes (determinism)** + +- Bundled recipe SQL adds stable secondary **`ORDER BY`** columns (and orders inner **`LIMIT`** samples) so **`--recipe`** / **`--json`** output does not vary on aggregate ties. + +**External QA** + +- **`bun run qa:external`**: **`--max-files`** and **`--max-symbols`** must be positive integers (invalid values throw before indexing). diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 8edc4bdf..333798e3 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -11,7 +11,9 @@ Codemap is in **bootstrap / extraction** phase. Before large PRs, please open an bun install # runs `prepare` → Husky git hooks bun run dev # same as `bun src/index.ts` — CLI from source bun test -bun run check # format + lint + tests + typecheck + build +bun run test:golden # golden SQL vs fixtures/minimal (also runs at end of `bun run check`) +bun run test:golden:external # Tier B: local tree via CODEMAP_ROOT / --root (not in CI) +bun run check # format + lint + tests + typecheck + build + test:golden bun run clean # remove untracked/ignored build artifacts (keeps `.env`, `.codemap.db*`) bun run check-updates # interactive dependency updates (`bun update -i --latest`) ``` @@ -36,6 +38,8 @@ Then open a PR on GitHub into **`main`**. - **Public API** — Anything exported from the package entry (`src/index.ts` → `src/api.ts`, `config.ts`, shared types) should have **JSDoc** that reads well in hovers and in published typings. - **Layers** — Keep boundaries clear: [architecture.md](../docs/architecture.md) (`cli` → `application` → infrastructure). Don’t let CLI concerns leak into parsers or the DB layer. - **Before you open / update a PR** — `bun run check` (or at least `bun run test` + `bun run typecheck` while iterating). +- **Golden queries (Tier A)** — If you change `fixtures/minimal/` or schema/query behavior expected by [fixtures/golden/](../fixtures/golden/), run `bun scripts/query-golden.ts --update`, review diffs, and commit updated JSON under `fixtures/golden/minimal/`. Prefer **fixing the indexer** when output changes for the wrong reason; only refresh goldens when the new rows are correct. See [docs/golden-queries.md](../docs/golden-queries.md). +- **Golden queries (Tier B)** — Against a **local** clone, use `bun run test:golden:external` with `CODEMAP_ROOT` / `--root`. Copy [fixtures/golden/scenarios.external.example.json](../fixtures/golden/scenarios.external.example.json) to `scenarios.external.json` if you need custom scenarios; goldens under `fixtures/golden/external/` are gitignored — do not commit snapshots from proprietary trees. - **Style** — Match Oxfmt/Oxlint; prefer **straight-line code** and extracted helpers over long nested blocks. **Editor (VS Code):** [`.vscode/extensions.json`](../.vscode/extensions.json) lists recommended extensions (Bun, Oxc, TypeScript native preview, etc.). [`.vscode/settings.json`](../.vscode/settings.json) enables Oxc format on save and `tsgo`. Formatting and lint rules live in [`.oxfmtrc.json`](../.oxfmtrc.json) and [`.oxlintrc.json`](../.oxlintrc.json) (no framework-specific options beyond defaults). @@ -46,6 +50,8 @@ Then open a PR on GitHub into **`main`**. Do **not** add Codemap as a dependency to the bench repo. In **this** repo, copy `.env.example` to `.env` and set **`CODEMAP_TEST_BENCH`** to an **absolute path** to the other clone, then run `bun src/index.ts` as usual. See [docs/benchmark.md § Indexing another project](../docs/benchmark.md#indexing-another-project). +**One-shot QA (index + disk checks + benchmark):** `CODEMAP_ROOT=/absolute/path/to/app bun run qa:external` (or set **`CODEMAP_TEST_BENCH`** in `.env`; optional `--root` overrides). Optional **`--max-files`** / **`--max-symbols`** (positive integers; default caps sampling). Validates indexed paths exist, spot-checks symbol lines vs files, prints sample SQL rows, then runs `src/benchmark.ts`. Do **not** add external app source into this repository. + Releases: **[@changesets/cli](https://github.com/changesets/changesets)** — run **`bun run changeset`** when your PR should bump the version; see [docs/packaging.md § Releases](../docs/packaging.md#releases). **Issues:** use [GitHub issue templates](https://github.com/stainless-code/codemap/issues/new/choose) — **Core bug** vs **Adapter proposal** (see `.github/ISSUE_TEMPLATE/`). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 84f1f228..0f156b79 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -92,6 +92,9 @@ jobs: - name: Run unit tests with coverage run: bun run test:coverage + - name: Golden query regression (fixtures/minimal) + run: bun run test:golden + build: name: 🧰 Build needs: skip-ci diff --git a/.gitignore b/.gitignore index 734a1dcd..10649298 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,11 @@ dist/ .env.* !.env.example tsconfig.lint-staged.json + +# Tier B golden query outputs (local / private trees — see docs/golden-queries.md) +fixtures/golden/external/ +fixtures/golden/scenarios.external.json + +# QA chat prompts tied to a private/local index (paths + product names) +fixtures/qa/*.local.md +fixtures/benchmark/*.local.json diff --git a/README.md b/README.md index 52a5974b..f8f3f9d0 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ - **Not** full-text search or grep on arbitrary strings — use those when you need raw file-body search. - **Is** a fast, token-efficient way to navigate **structure**: definitions, imports, dependency direction, components, and other extracted facts. -**Documentation:** [docs/README.md](docs/README.md) is the hub (topic index + single-source rules). Topics: [architecture](docs/architecture.md), [agents](docs/agents.md) (`codemap agents init`), [benchmark](docs/benchmark.md), [packaging](docs/packaging.md), [roadmap](docs/roadmap.md), [why Codemap](docs/why-codemap.md). **Bundled rules/skills:** [`.agents/rules/`](.agents/rules/), [`.agents/skills/codemap/SKILL.md`](.agents/skills/codemap/SKILL.md). **Consumers:** [.github/CONTRIBUTING.md](.github/CONTRIBUTING.md). +**Documentation:** [docs/README.md](docs/README.md) is the hub (topic index + single-source rules). Topics: [architecture](docs/architecture.md), [agents](docs/agents.md) (`codemap agents init`), [benchmark](docs/benchmark.md), [golden queries](docs/golden-queries.md), [packaging](docs/packaging.md), [roadmap](docs/roadmap.md), [why Codemap](docs/why-codemap.md). **Bundled rules/skills:** [`.agents/rules/`](.agents/rules/), [`.agents/skills/codemap/SKILL.md`](.agents/skills/codemap/SKILL.md). **Consumers:** [.github/CONTRIBUTING.md](.github/CONTRIBUTING.md). --- @@ -37,6 +37,16 @@ codemap --full # SQL against the index (after at least one index run) codemap query "SELECT name, file_path FROM symbols LIMIT 10" +# JSON array on stdout (agents / scripts); {"error":"..."} for bad SQL, DB open, or query bootstrap (config/resolver) when using --json +codemap query --json "SELECT name, file_path FROM symbols LIMIT 10" +# Query is not row-capped — add LIMIT in SQL for large selects +# Bundled SQL (same as skill examples): fan-out rankings +codemap query --recipe fan-out +codemap query --json --recipe fan-out-sample +# List bundled recipes as JSON, or print one recipe's SQL (no DB required) +codemap query --recipes-json +codemap query --print-sql fan-out +# `components-by-hooks` ranks by hook count without SQLite JSON1 (comma-based count on the stored JSON array). # Another project codemap --root /path/to/repo --full @@ -81,12 +91,15 @@ const rows = cm.query("SELECT name FROM symbols LIMIT 5"); Tooling: **Oxfmt**, **Oxlint**, **tsgo** (`@typescript/native-preview`). -| Command | Purpose | -| ------------------------------------ | ---------------------------------------------------------------- | -| `bun run dev` | Run the CLI from source (same as `bun src/index.ts`) | -| `bun run check` | Build, format check, lint, tests, typecheck — run before pushing | -| `bun run fix` | Apply lint fixes, then format | -| `bun run test` / `bun run typecheck` | Focused checks | +| Command | Purpose | +| ------------------------------------ | -------------------------------------------------------------------------- | +| `bun run dev` | Run the CLI from source (same as `bun src/index.ts`) | +| `bun run check` | Build, format check, lint, tests, typecheck — run before pushing | +| `bun run fix` | Apply lint fixes, then format | +| `bun run test` / `bun run typecheck` | Focused checks | +| `bun run test:golden` | SQL snapshot regression on `fixtures/minimal` (included in `check`) | +| `bun run test:golden:external` | Tier B: local tree via `CODEMAP_*` / `--root` (not in default `check`) | +| `bun run qa:external` | Index + sanity checks + benchmark on `CODEMAP_ROOT` / `CODEMAP_TEST_BENCH` | ```bash bun install @@ -100,11 +113,13 @@ bun run fix # oxlint --fix, then oxfmt ## Benchmark +Use a **real** project path (the repo must exist on disk). See [docs/benchmark.md § Indexing another project](docs/benchmark.md#indexing-another-project). + ```bash -CODEMAP_ROOT=/path/to/indexed-repo bun src/benchmark.ts +CODEMAP_ROOT=/absolute/path/to/indexed-repo bun src/benchmark.ts ``` -Details: [docs/benchmark.md](docs/benchmark.md). +Optional **`CODEMAP_BENCHMARK_CONFIG`** for repo-specific scenarios: [docs/benchmark.md § Custom scenarios](docs/benchmark.md#custom-scenarios-codemap_benchmark_config). --- diff --git a/docs/README.md b/docs/README.md index 6124d128..2eb32213 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,28 +4,36 @@ Technical docs for **[@stainless-code/codemap](https://github.com/stainless-code **Start here:** [../README.md](../README.md) (install, CLI, API, dev commands). **This folder** is deeper reference — pick a row below. -| File | Topic | -| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [why-codemap.md](./why-codemap.md) | Why index + SQL for agents (speed, tokens, accuracy) — good first read after the readme | -| [architecture.md](./architecture.md) | Schema, layering, CLI internals, API, [**User config**](./architecture.md#user-config) (Zod), parsers, [Key Files](./architecture.md#key-files) | -| [agents.md](./agents.md) | **`codemap agents init`** — bundled **`templates/agents`** → **`.agents/`** in **consumer projects** (this repo’s **`.agents/`** is dev/maintainer — see [agents.md](./agents.md)); per-file IDE symlink/copy, **[pointer files](./agents.md#pointer-files)** (`codemap-pointer`), **`--interactive`**, **`.gitignore` / `.codemap.*`** | -| [benchmark.md](./benchmark.md) | [**Indexing another project**](./benchmark.md#indexing-another-project) · [**Benchmark script**](./benchmark.md#the-benchmark-script) · [`fixtures/minimal/`](../fixtures/minimal/) | -| [packaging.md](./packaging.md) | **`CHANGELOG.md` / `dist/` / `templates/`** on npm, **engines**, [**Node vs Bun**](./packaging.md#node-vs-bun), [**Releases**](./packaging.md#releases) (Changesets) | -| [roadmap.md](./roadmap.md) | Forward-looking backlog (not a `src/` inventory) | +| File | Topic | +| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [why-codemap.md](./why-codemap.md) | Why index + SQL for agents (speed, tokens, accuracy) — good first read after the readme | +| [architecture.md](./architecture.md) | Schema, layering, CLI internals, API, [**User config**](./architecture.md#user-config) (Zod), parsers, [Key Files](./architecture.md#key-files) | +| [agents.md](./agents.md) | **`codemap agents init`** — bundled **`templates/agents`** → **`.agents/`** in **consumer projects** (this repo’s **`.agents/`** is dev/maintainer — see [agents.md](./agents.md)); per-file IDE symlink/copy, **[pointer files](./agents.md#pointer-files)** (`codemap-pointer`), **`--interactive`**, **`.gitignore` / `.codemap.*`** | +| [benchmark.md](./benchmark.md) | [**Indexing another project**](./benchmark.md#indexing-another-project) · [**Benchmark script**](./benchmark.md#the-benchmark-script) · [**Custom scenarios**](./benchmark.md#custom-scenarios-codemap_benchmark_config) (`CODEMAP_BENCHMARK_CONFIG`) · [`fixtures/minimal/`](../fixtures/minimal/) | +| [golden-queries.md](./golden-queries.md) | Golden `query` **design & policy** (Tier A/B, no proprietary trees); runner: [scripts/query-golden.ts](../scripts/query-golden.ts) | +| [fixtures/golden/](../fixtures/golden/) | [scenarios.json](../fixtures/golden/scenarios.json) + [minimal/](../fixtures/golden/minimal/) — **`bun run test:golden`**; Tier B: [scenarios.external.example.json](../fixtures/golden/scenarios.external.example.json) + **`bun run test:golden:external`** ([benchmark § Fixtures](./benchmark.md#fixtures)) | +| [fixtures/benchmark/](../fixtures/benchmark/) | Tracked [scenarios.example.json](../fixtures/benchmark/scenarios.example.json) — copy to `*.local.json` (gitignored) for [`CODEMAP_BENCHMARK_CONFIG`](./benchmark.md#custom-scenarios-codemap_benchmark_config) | +| [fixtures/qa/](../fixtures/qa/) | [prompts.external.template.md](../fixtures/qa/prompts.external.template.md) — optional chat QA prompts for an external index (`*.local.md` gitignored) | +| [packaging.md](./packaging.md) | **`CHANGELOG.md` / `dist/` / `templates/`** on npm, **engines**, [**Node vs Bun**](./packaging.md#node-vs-bun), [**Releases**](./packaging.md#releases) (Changesets) | +| [roadmap.md](./roadmap.md) | Forward-looking backlog (not a `src/` inventory) | ## Single source of truth (do not duplicate) -| Topic | Canonical doc | Elsewhere | -| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | -| Runtime splits (SQLite, workers, globs, JSON config I/O) | [packaging § Node vs Bun](./packaging.md#node-vs-bun) — **the table lives here** | [architecture § Runtime](./architecture.md#runtime-and-database) links here; do not copy the table | -| **`codemap.config.*`** shape / Zod validation | [architecture § User config](./architecture.md#user-config) | Root [README § Configuration](../README.md#configuration) points here | -| **`codemap agents init`**: **`--force`** on **`.agents/`** in **consumer projects** (template file paths only), IDE matrix, per-file symlink/copy, **`templates/agents`** | [agents.md](./agents.md) | Link here; do not paste the integration table into README or packaging | -| **`CLAUDE.md` / `AGENTS.md` / `GEMINI.md` / Copilot** — managed **`codemap-pointer`** sections, merge vs **`--force`** | [agents.md § Pointer files](./agents.md#pointer-files) | Link here; do not duplicate the situation table | -| End-user CLI (index, query, agents, flags, env) | [../README.md § CLI](../README.md#cli) | [architecture § CLI usage](./architecture.md#cli-usage) summarizes and links back | +| Topic | Canonical doc | Elsewhere | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| Runtime splits (SQLite, workers, globs, JSON config I/O) | [packaging § Node vs Bun](./packaging.md#node-vs-bun) — **the table lives here** | [architecture § Runtime](./architecture.md#runtime-and-database) links here; do not copy the table | +| **`codemap.config.*`** shape / Zod validation | [architecture § User config](./architecture.md#user-config) | Root [README § Configuration](../README.md#configuration) points here | +| **`codemap agents init`**: **`--force`** on **`.agents/`** in **consumer projects** (template file paths only), IDE matrix, per-file symlink/copy, **`templates/agents`** | [agents.md](./agents.md) | Link here; do not paste the integration table into README or packaging | +| **`CLAUDE.md` / `AGENTS.md` / `GEMINI.md` / Copilot** — managed **`codemap-pointer`** sections, merge vs **`--force`** | [agents.md § Pointer files](./agents.md#pointer-files) | Link here; do not duplicate the situation table | +| End-user CLI (index, **`query --json`**, **`query --recipe`**, **`query --recipes-json`**, **`query --print-sql`**, agents, flags, env) — query has no row cap; use SQL **`LIMIT`**; **`--json`** errors include SQL, DB open, and bootstrap failures | [../README.md § CLI](../README.md#cli) | [architecture § CLI usage](./architecture.md#cli-usage) summarizes and links back | +| Golden query regression (`test:golden`, `test:golden:external`, `--update`) | [golden-queries.md](./golden-queries.md) | CONTRIBUTING § Golden queries; [benchmark § Fixtures](./benchmark.md#fixtures) | +| **`CODEMAP_BENCHMARK_CONFIG`** (per-repo benchmark JSON) | [benchmark § Custom scenarios](./benchmark.md#custom-scenarios-codemap_benchmark_config) | [fixtures/benchmark/scenarios.example.json](../fixtures/benchmark/scenarios.example.json) only | +| `bun run qa:external` — index + disk checks + `benchmark.ts` on **`CODEMAP_*`** | [.github/CONTRIBUTING.md](../.github/CONTRIBUTING.md) | [scripts/qa-external-repo.ts](../scripts/qa-external-repo.ts) (invocation only) | ## Conventions - **One topic per file**; prefer relative links between these docs. +- **CLI flags and examples** — canonical [README.md § CLI](../README.md#cli). Other docs **summarize and link**; do not copy full flag lists (see **Single source of truth** above). **Implementation paths** (`src/cli/…`, **`QUERY_RECIPES`**) belong in [architecture.md § CLI usage](./architecture.md#cli-usage) only. - **Avoid stale file/symbol counts** in narrative text — use `codemap query` / `bun run dev query` after indexing; methodology tables in [benchmark.md](./benchmark.md) are fine. - **This repo:** `bun run dev` is **`bun src/index.ts`**; `bun run build` → tsdown → `dist/`; `bun run clean` / `bun run check-updates` — [.github/CONTRIBUTING.md](../.github/CONTRIBUTING.md). - **Contributors:** branch + PR into **`main`** ([CI](../.github/workflows/ci.yml)), `bun run check`, JSDoc on public API. diff --git a/docs/architecture.md b/docs/architecture.md index c067e730..99eb5d7c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -92,30 +92,32 @@ A local SQLite database (`.codemap.db`) indexes the project tree and stores stru ## Key Files -| File | Purpose | -| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -| `index.ts` | Package entry — re-exports `api` / `config`, runs CLI when main | -| `cli/` | CLI — bootstrap argv, lazy command modules, `query` / `agents init` / index modes | -| `api.ts` | Programmatic API — `createCodemap`, `Codemap`, `runCodemapIndex` | -| `application/` | Indexing use cases and engine (`run-index`, `index-engine`, types) | -| `worker-pool.ts` | Parallel parse workers (Bun / Node) | -| `db.ts` | SQLite adapter — schema DDL, typed CRUD, connection management | -| `parser.ts` | TS/TSX/JS/JSX extraction via `oxc-parser` — symbols, imports, exports, components, markers | -| `css-parser.ts` | CSS extraction via `lightningcss` — custom properties, classes, keyframes, `@theme` blocks | -| `resolver.ts` | Import path resolution via `oxc-resolver` — respects `tsconfig` aliases, builds dependency graph | -| `constants.ts` | Shared constants — e.g. `LANG_MAP` | -| `glob-sync.ts` | Include globs — Bun `Glob` vs `fast-glob` on Node ([packaging § Node vs Bun](./packaging.md#node-vs-bun)) | -| `markers.ts` | Shared marker extraction (`TODO`/`FIXME`/`HACK`/`NOTE`) — used by all parsers | -| `parse-worker.ts` | Worker thread entry point — reads, parses, and extracts file data in parallel | -| `adapters/` | `LanguageAdapter` types and built-in TS/CSS/text implementations | -| `parsed-types.ts` | Shared `ParsedFile` shape for workers and adapters | -| `agents-init.ts` / `agents-init-interactive.ts` | `codemap agents init` — see [agents.md](./agents.md) (granular template + IDE writes, pointer upsert, **`--interactive`**, `.gitignore`) | -| `benchmark.ts` | SQL vs traditional timing script — see [benchmark.md § The benchmark script](./benchmark.md#the-benchmark-script) | -| `config.ts` | `codemap.config.*` load path, **Zod** user schema (`codemapUserConfigSchema`), `resolveCodemapConfig` | +| File | Purpose | +| ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `index.ts` | Package entry — re-exports `api` / `config`, runs CLI when main | +| `cli/` | CLI — bootstrap argv, lazy command modules, `query` / `agents init` / index modes | +| `api.ts` | Programmatic API — `createCodemap`, `Codemap`, `runCodemapIndex` | +| `application/` | Indexing use cases and engine (`run-index`, `index-engine`, types) | +| `worker-pool.ts` | Parallel parse workers (Bun / Node) | +| `db.ts` | SQLite adapter — schema DDL, typed CRUD, connection management | +| `parser.ts` | TS/TSX/JS/JSX extraction via `oxc-parser` — symbols, imports, exports, components, markers | +| `css-parser.ts` | CSS extraction via `lightningcss` — custom properties, classes, keyframes, `@theme` blocks | +| `resolver.ts` | Import path resolution via `oxc-resolver` — respects `tsconfig` aliases, builds dependency graph | +| `constants.ts` | Shared constants — e.g. `LANG_MAP` | +| `glob-sync.ts` | Include globs — Bun `Glob` vs `fast-glob` on Node ([packaging § Node vs Bun](./packaging.md#node-vs-bun)) | +| `markers.ts` | Shared marker extraction (`TODO`/`FIXME`/`HACK`/`NOTE`) — used by all parsers | +| `parse-worker.ts` | Worker thread entry point — reads, parses, and extracts file data in parallel | +| `adapters/` | `LanguageAdapter` types and built-in TS/CSS/text implementations | +| `parsed-types.ts` | Shared `ParsedFile` shape for workers and adapters | +| `agents-init.ts` / `agents-init-interactive.ts` | `codemap agents init` — see [agents.md](./agents.md) (granular template + IDE writes, pointer upsert, **`--interactive`**, `.gitignore`) | +| `benchmark.ts` (+ `benchmark-default-scenarios.ts`, `benchmark-config.ts`, `benchmark-common.ts`) | SQL vs traditional timing; optional **`CODEMAP_BENCHMARK_CONFIG`** JSON — [benchmark.md § Custom scenarios](./benchmark.md#custom-scenarios-codemap_benchmark_config) | +| `config.ts` | `codemap.config.*` load path, **Zod** user schema (`codemapUserConfigSchema`), `resolveCodemapConfig` | ## CLI usage -**Commands and flags** (index, query, **`codemap agents init`**, **`--root`**, **`--config`**, environment): [../README.md § CLI](../README.md#cli). From this repository: **`bun run dev`** or **`bun src/index.ts`** (same flags). +**Commands and flags** (index, query, **`codemap agents init`**, **`--root`**, **`--config`**, environment): [../README.md § CLI](../README.md#cli) — **do not duplicate** flag lists here; this section only adds implementation notes. From this repository: **`bun run dev`** or **`bun src/index.ts`** (same flags). + +**Query wiring:** **`src/cli/cmd-query.ts`** (argv, **`printQueryResult`**), **`src/cli/query-recipes.ts`** (**`QUERY_RECIPES`** — bundled SQL only source), **`src/cli/main.ts`** (**`--recipes-json`** / **`--print-sql`** exit before config/DB). With **`--json`**, errors use **`{"error":"…"}`** on stdout for SQL failures, DB open, and bootstrap (same shape); **`runQueryCmd`** sets **`process.exitCode`** instead of **`process.exit`**. The **`components-by-hooks`** recipe ranks by hook count with a **comma-based tally** on **`hooks_used`** (no SQLite JSON1). **Agent templates:** `codemap agents init` — full matrix [agents.md](./agents.md). diff --git a/docs/benchmark.md b/docs/benchmark.md index 085cbb5a..a3e53697 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -71,18 +71,34 @@ bun src/benchmark.ts bun src/benchmark.ts --verbose ``` +### Custom scenarios (`CODEMAP_BENCHMARK_CONFIG`) + +For a **specific** checkout (e.g. large app), you can replace or extend the eight built-in demo scenarios with JSON so the **Results** column lines up with **indexed SQL** vs **glob + regex** on the same corpus. + +```bash +# Use a real checkout path — /path/to/repo in docs is a placeholder only. +# Resolve CODEMAP_BENCHMARK_CONFIG from the shell cwd (see below). +CODEMAP_ROOT=/absolute/path/to/your-app CODEMAP_BENCHMARK_CONFIG=fixtures/benchmark/my.local.json bun src/benchmark.ts +``` + +- **Tracked example:** [fixtures/benchmark/scenarios.example.json](../fixtures/benchmark/scenarios.example.json) — copy to `*.local.json` (see [.gitignore](../.gitignore); do not commit proprietary paths). +- **`CODEMAP_BENCHMARK_CONFIG`** is passed to **`path.resolve()`** from **`process.cwd()`** — use an absolute path or a path relative to where you run the command (not relative to **`src/`**). +- Each entry has **`name`**, **`indexedSql`**, and **`traditional`**: either **`{ "globs": [...], "regex": "...", "mode": "files" | "matches" }`** or **`{ "builtin": "fanoutImportLines" }`** (same traditional path as **`--recipe fan-out`**). **`indexedSql`** must be a **single** read-only **`SELECT`** (or **`WITH` … `SELECT`**) — mutating statements are **rejected** at load time. +- **`traditional.regex`:** treated as trusted input from your local JSON (benchmark tooling is developer-facing). **`mode": "files"`** reuses one **`RegExp`** per scenario for efficiency. +- **`replaceDefault`:** `true` (default) uses only this list; `false` **appends** these scenarios after the built-in eight. + ### Methodology Each scenario runs both approaches back-to-back on the same machine, same data. Measured: -| Metric | What it captures | -| ---------- | ----------------------------------------------------------------- | -| Index Time | Wall-clock time for the SQL query | -| Trad. Time | Wall-clock time for glob + read all matching files + regex search | -| Results | Number of matches returned | -| Files Read | How many files the traditional approach had to read | -| Bytes Read | Total source bytes loaded into memory by the traditional approach | -| Speedup | `traditionalMs / indexMs` | +| Metric | What it captures | +| ---------- | -------------------------------------------------------------------------------------------- | +| Index Time | Wall-clock time for the SQL query | +| Trad. Time | Wall-clock time for glob + read all matching files + regex search | +| Results | Number of matches returned | +| Files Read | How many **unique** files the traditional approach read (overlapping globs are deduplicated) | +| Bytes Read | Total source bytes loaded for those unique paths (each file counted once) | +| Speedup | `traditionalMs / indexMs` | **Important**: the traditional approach simulates best-case AI tool behavior — it reads files in-process with Bun's fast I/O. Real AI agent tool calls add network round-trips, context window serialization, and multiple turn overhead that make the gap significantly larger. @@ -97,15 +113,23 @@ This document measures **indexed SQL vs traditional glob/read** on an existing d ### Scenarios -| # | Scenario | What it tests | -| --- | --------------------------------------- | ---------------------------------------------------- | -| 1 | Find where `usePermissions` is defined | Symbol lookup by name — needle in haystack | -| 2 | List React components (TSX/JSX) | AST `components` table vs export-line regex | -| 3 | Files that import from `~/api/client` | Large result set — LIKE scan vs grep | -| 4 | Find all TODO/FIXME markers | Cross-file scan — all file types | -| 5 | CSS design tokens (custom properties) | Domain-specific extraction — structured vs raw regex | -| 6 | Components in `shop/` subtree | Scoped component discovery | -| 7 | Reverse deps: who imports `utils/date`? | Dependency graph traversal | +| # | Scenario | What it tests | +| --- | --------------------------------------- | ------------------------------------------------------------------- | +| 1 | Find where `usePermissions` is defined | Symbol lookup by name — needle in haystack | +| 2 | List React components (TSX/JSX) | AST `components` table vs export-line regex | +| 3 | Files that import from `~/api/client` | Large result set — LIKE scan vs grep | +| 4 | Find all TODO/FIXME markers | Cross-file scan — all file types | +| 5 | CSS design tokens (custom properties) | Domain-specific extraction — structured vs raw regex | +| 6 | Components in `shop/` subtree | Scoped component discovery | +| 7 | Reverse deps: who imports `utils/date`? | Dependency graph traversal | +| 8 | Top 10 by dependency fan-out | Same SQL as **`codemap query --recipe fan-out`** vs line-scan proxy | + +### Recipes vs ad-hoc SQL + +**`codemap query --recipe `** expands to the same SQL as pasting that string after **`codemap query`** ([README § CLI](../README.md#cli) — **`--print-sql`**, **`--recipes-json`**). There is no extra query cost beyond parsing argv — **recipe vs hand-written SQL is not a separate benchmark**. + +- **`fan-out-sample`** uses **`GROUP_CONCAT`** for sample targets (portable). +- **`fan-out-sample-json`** uses **`json_group_array`** in the same shape (requires SQLite JSON1). Prefer **`fan-out-sample`** when JSON1 is unavailable. ### Results @@ -120,8 +144,9 @@ Example snapshot from `bun src/benchmark.ts` immediately after `bun src/index.ts | CSS design tokens (custom properties) | 47µs | 0 | 2.78ms | 0 | 0 | 0 B | **59×** | | Components in `shop/` subtree | 40µs | 0 | 2.61ms | 0 | 0 | 0 B | **66×** | | Reverse deps: who imports `utils/date`? | 39µs | 0 | 3.59ms | 0 | 13 | 76.3 KB | **93×** | +| Top 10 by dependency fan-out | 81µs | 2 | 15.24ms | 10 | 56 | 163.4 KB | **188×** | -**Totals**: Index ~408µs vs Traditional ~26.7ms (**~65× overall** on a sample run). Traditional bytes read total ~393 KB (not megabytes) because the globbed sets are small. +**Totals** (8 scenarios; sample run on this repo after adding scenario 8): Index ~1.1ms vs Traditional ~140ms (**~132× overall**). Traditional bytes read total ~940 KB on that run — your tree and hardware will differ. Older 7-scenario snapshots showed ~408µs / ~27ms / ~393 KB. On a **large app** indexed via `--root`, the same queries typically return non-zero rows; the indexed side stays sub-millisecond while the traditional side reads megabytes for broad globs. Repeatable numbers: [Fixtures](#fixtures). @@ -131,7 +156,7 @@ On a small repo, totals move with noise and thermal variance. On a large indexed The script’s **reindex** section averages **3 internal runs** per mode; full-rebuild wall time varies with disk and CPU load. -The indexed CSS scenario uses `ORDER BY name LIMIT 50` — exact SQL for each scenario lives in **`src/benchmark.ts`** in this repo (not duplicated here; keep in sync when changing scenarios). +The indexed CSS scenario uses `ORDER BY name LIMIT 50`. The **fan-out** row’s indexed path uses **`getQueryRecipeSql("fan-out")`** from **`src/cli/query-recipes.ts`** (same text as **`codemap query --recipe fan-out`**). Other default scenarios’ SQL lives in **`src/benchmark-default-scenarios.ts`**; custom JSON is loaded in **`src/benchmark-config.ts`** (keep **`fixtures/benchmark/scenarios.example.json`** in sync when recipe SQL changes). ### Key takeaways @@ -184,4 +209,8 @@ bun run benchmark **CI:** the workflow **Benchmark (fixture)** runs the same steps with `CODEMAP_ROOT=$GITHUB_WORKSPACE/fixtures/minimal`. +**Correctness (golden queries):** `bun run test:golden` indexes `fixtures/minimal`, runs SQL against [fixtures/golden/scenarios.json](../fixtures/golden/scenarios.json), and compares to [fixtures/golden/minimal/](../fixtures/golden/minimal/). See [golden-queries.md](./golden-queries.md). Refresh goldens after intentional fixture or schema changes: `bun scripts/query-golden.ts --update`. + +**Tier B (local tree, not in default CI):** `bun run test:golden:external` (or `bun scripts/query-golden.ts --corpus external`) indexes **`CODEMAP_ROOT`**, **`CODEMAP_TEST_BENCH`**, or **`--root`**, loads [fixtures/golden/scenarios.external.json](../fixtures/golden/scenarios.external.json) if present else [scenarios.external.example.json](../fixtures/golden/scenarios.external.example.json), and writes/compares goldens under `fixtures/golden/external/` (gitignored). Use **`match`** in scenarios for subset checks (`minRows`, `everyRowContains`); use **`budgetMs`** with optional **`--strict-budget`** for perf warnings. Do not commit proprietary paths or goldens from private apps. + Scenario titles match the table above; **indexed row counts** on the fixture are stable for a given schema. A larger second fixture is optional — see [roadmap.md](./roadmap.md). diff --git a/docs/golden-queries.md b/docs/golden-queries.md new file mode 100644 index 00000000..867dbff9 --- /dev/null +++ b/docs/golden-queries.md @@ -0,0 +1,85 @@ +# Golden queries — design & policy + +**Purpose:** Regression-test **Codemap internals** by comparing **`codemap query`** output to **checked-in expectations** (or subset matchers) on fixed corpora — **not** an LLM-in-the-loop eval. **Latency / tokens vs scanning:** [benchmark.md](./benchmark.md). + +**Operational docs:** [CONTRIBUTING § Golden queries](../.github/CONTRIBUTING.md) · [benchmark § Fixtures](./benchmark.md#fixtures) · Runner: [scripts/query-golden.ts](../scripts/query-golden.ts) · Schema: [scripts/query-golden/schema.ts](../scripts/query-golden/schema.ts) + +--- + +## Goals + +| Goal | How scenarios help | +| ------------------------- | ------------------------------------------------ | +| **Catch regressions** | Parser or schema drift → JSON diff vs golden | +| **Encode good answers** | Human-reviewed rows for representative queries | +| **Stress realistic size** | Optional second corpus beyond `fixtures/minimal` | +| **Stay deterministic** | Assertions on **query output**, not model prose | + +## Non-goals + +- **Chat / SSE / auth** harnesses — out of scope here +- **Proving agents follow rules** — measure in the IDE or another project +- **Replacing** `src/benchmark.ts` — that stays **SQL vs glob/read time**; goldens add **correctness snapshots** + +--- + +## How this fits other tooling + +| Piece | Role | +| ------------------------- | ------------------------------------------------------------ | +| `fixtures/minimal/` | Tier **A** corpus; stable for CI | +| `src/benchmark.ts` | Speed comparison (not golden row equality) | +| `bun test` | Unit tests for parsers, CLI, DB | +| `CODEMAP_ROOT` / `--root` | Index **any** tree; Tier **B** uses env + optional gitignore | + +--- + +## No proprietary app code in this repo + +We **do not** commit another product’s source tree, paths, business strings, or golden JSON **derived from** a private app (or any repo we do not own and license for redistribution). + +| Safe to commit here | Not committed here | +| ------------------------------------------ | ------------------------------------ | +| **`fixtures/minimal/`** (trees we control) | Clones of private apps | +| **Generic SQL** / **`--recipe`** ids | App-specific path literals in assets | +| **Goldens** from **our** fixtures only | Snapshots keyed to proprietary names | +| **Abstract `prompt` text** (intent labels) | Verbatim customer prompts | + +**Tier B:** Point `CODEMAP_ROOT` at a **local** clone; goldens for that tree stay **gitignored** (or private automation) — see [.gitignore](../.gitignore) and [benchmark § Tier B](./benchmark.md#fixtures). + +--- + +## Tier model + +| Tier | Corpus | When | Purpose | +| --------------- | ---------------------------- | -------------------------- | ---------------------------------------- | +| **A** | `fixtures/minimal` (in-repo) | Every PR / `bun run check` | Fast, **committed** goldens | +| **B** | Local path via `CODEMAP_*` | Maintainer machine | Scale; goldens **optional / gitignored** | +| **B′** (future) | Public OSS fixture only | CI optional | Larger committed corpus if license OK | + +--- + +## Scenario shape (implemented) + +Scenarios live in **`fixtures/golden/scenarios.json`** (Tier A) or optional **`scenarios.external.json`** / **example** (Tier B). Each entry has **`id`**, **`sql` or `recipe`**, optional **`match`** (`exact`, `minRows`, `everyRowContains`), optional **`budgetMs`**. Goldens: **`fixtures/golden/minimal/*.json`** etc. Refresh: **`bun scripts/query-golden.ts --update`**. + +**Prompts** in JSON are **intent labels**, not pasted chat logs — pair with queries whose literals come from **fixture-owned** data (see [fixtures/qa/prompts.external.template.md](../fixtures/qa/prompts.external.template.md) for optional chat QA). + +--- + +## Status + +| Area | State | +| ----------------------------- | ----------------------------------------------------------------------- | +| Tier A runner + CI | **`bun run test:golden`** in `check` | +| Tier B external + schema | **`test:golden:external`**, Zod in **`scripts/query-golden/schema.ts`** | +| Subset matchers + budgets | **`match`**, **`budgetMs`**, **`--strict-budget`** | +| Optional CI for public corpus | Deferred — [roadmap § Backlog](./roadmap.md#backlog) | + +--- + +## References + +- [benchmark.md](./benchmark.md) — speed methodology, Tier B, fixtures +- [architecture.md](./architecture.md) — schema, parsers +- [roadmap.md](./roadmap.md) — backlog diff --git a/docs/packaging.md b/docs/packaging.md index 9250cd5b..3a02ac2a 100644 --- a/docs/packaging.md +++ b/docs/packaging.md @@ -38,6 +38,8 @@ Releases use [**Changesets**](https://github.com/changesets/changesets). Repo co | 2 | **Merge** — on every push to **`main`**, [`.github/workflows/release.yml`](../.github/workflows/release.yml) runs [`changesets/action@v1`](https://github.com/changesets/action): opens/updates the **Version packages** PR when pending changesets exist; **`publish: bun run release`** runs **`changeset publish`**; **`createGithubReleases: true`**. | | 3 | **Secrets** — **`GITHUB_TOKEN`** is provided by Actions. **`NPM_TOKEN`** (npm [automation token](https://docs.npmjs.com/creating-and-viewing-access-tokens)) must be a **repository secret** for publishes to npm. If the Release job fails, use the workflow log (missing token, registry error, etc.) — don’t assume the cause from the job name alone. | +**Before tagging / publishing:** run **`bun run check`** on the branch that will ship. Pending **`.changeset/*.md`** files are turned into **`CHANGELOG.md`** entries and a version bump by the **Version packages** PR (do not hand-edit **`package.json`** version for Changesets-driven releases). Merge that PR to **`main`** to run **`changeset publish`** via the workflow. + ## Related - [architecture.md](./architecture.md) — schema, layering, API, user config. diff --git a/docs/roadmap.md b/docs/roadmap.md index 75979aa7..943c51ef 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -8,6 +8,7 @@ Forward-looking plans only — **not** a mirror of `src/`. **Doc index:** [READM - **Community language adapters** — optional packages (e.g. Tree-sitter) with a **peerDependency** on `@stainless-code/codemap` and a public **registration** API beyond built-ins in [`src/adapters/`](../src/adapters/). - **Agent tooling** — evaluate [TanStack Intent](https://tanstack.com/intent/latest/docs/overview) for versioned skills in `node_modules` (optional; **`codemap agents init`** remains the default). +- **Golden queries** — design & policy: [golden-queries.md](./golden-queries.md); Tier A in CI, Tier B via `CODEMAP_*` (see [benchmark § Fixtures](./benchmark.md#fixtures)). --- @@ -29,6 +30,7 @@ Forward-looking plans only — **not** a mirror of `src/`. **Doc index:** [READM ## Backlog +- [ ] Optional **GitHub Actions** `workflow_dispatch` — run golden/benchmark against a **public** corpus only (never private app code) - [ ] Optional **FTS5** for opt-in full-text - [ ] **MCP** server wrapping `query` - [ ] **Watch mode** for dev diff --git a/docs/why-codemap.md b/docs/why-codemap.md index 0c62185d..99fc065d 100644 --- a/docs/why-codemap.md +++ b/docs/why-codemap.md @@ -17,6 +17,10 @@ This burns context window, wastes tokens, slows response time, and produces less A pre-built SQLite index (`.codemap.db`) that extracts and structures code metadata at index time. Agents query it with SQL instead of scanning files. Timings, scenarios, and methodology: [benchmark.md](./benchmark.md). +## Bundled CLI recipes + +**Shipped SQL** for fan-out, fan-in, index stats, markers, components, etc. — **`codemap query --recipe`**, **`--json`**, **`--recipes-json`**, **`--print-sql`**, and examples: [README.md § CLI](../README.md#cli) (canonical). **`fan-out-sample`** vs **`fan-out-sample-json`**: same ranking; JSON1 vs **`GROUP_CONCAT`** — see readme or **`codemap query --help`**. Agent-oriented SQL and schema: bundled [**`SKILL.md`**](../.agents/skills/codemap/SKILL.md) (this repo) / **`codemap agents init`** templates for consumers. + ## Speed Gains ### Headline pattern diff --git a/fixtures/benchmark/scenarios.example.json b/fixtures/benchmark/scenarios.example.json new file mode 100644 index 00000000..07ca0c9c --- /dev/null +++ b/fixtures/benchmark/scenarios.example.json @@ -0,0 +1,19 @@ +{ + "replaceDefault": true, + "scenarios": [ + { + "name": "usePermissions definition (sample alignment)", + "indexedSql": "SELECT file_path, kind FROM symbols WHERE name = 'usePermissions'", + "traditional": { + "globs": ["**/*.{ts,tsx}"], + "regex": "export\\s+function\\s+usePermissions", + "mode": "files" + } + }, + { + "name": "Top 10 dependency fan-out (same as --recipe fan-out)", + "indexedSql": "SELECT from_path, COUNT(*) AS deps FROM dependencies GROUP BY from_path ORDER BY deps DESC, from_path ASC LIMIT 10", + "traditional": { "builtin": "fanoutImportLines" } + } + ] +} diff --git a/fixtures/golden/minimal/dependencies-from-consumer.json b/fixtures/golden/minimal/dependencies-from-consumer.json new file mode 100644 index 00000000..9aecb55e --- /dev/null +++ b/fixtures/golden/minimal/dependencies-from-consumer.json @@ -0,0 +1,10 @@ +[ + { + "from_path": "src/consumer.ts", + "to_path": "src/api/client.ts" + }, + { + "from_path": "src/consumer.ts", + "to_path": "src/utils/date.ts" + } +] diff --git a/fixtures/golden/minimal/files-count.json b/fixtures/golden/minimal/files-count.json new file mode 100644 index 00000000..4a98986c --- /dev/null +++ b/fixtures/golden/minimal/files-count.json @@ -0,0 +1,5 @@ +[ + { + "n": 10 + } +] diff --git a/fixtures/golden/minimal/imports-consumer-alias.json b/fixtures/golden/minimal/imports-consumer-alias.json new file mode 100644 index 00000000..dd823093 --- /dev/null +++ b/fixtures/golden/minimal/imports-consumer-alias.json @@ -0,0 +1,8 @@ +[ + { + "file_path": "src/consumer.ts", + "source": "~/api/client", + "specifiers": "[\"createClient\"]", + "is_type_only": 0 + } +] diff --git a/fixtures/golden/minimal/index-summary.json b/fixtures/golden/minimal/index-summary.json new file mode 100644 index 00000000..fc688241 --- /dev/null +++ b/fixtures/golden/minimal/index-summary.json @@ -0,0 +1,9 @@ +[ + { + "files": 10, + "symbols": 5, + "imports": 2, + "components": 1, + "dependencies": 2 + } +] diff --git a/fixtures/golden/minimal/markers-notes-todo.json b/fixtures/golden/minimal/markers-notes-todo.json new file mode 100644 index 00000000..619001d7 --- /dev/null +++ b/fixtures/golden/minimal/markers-notes-todo.json @@ -0,0 +1,8 @@ +[ + { + "file_path": "src/notes.md", + "line_number": 3, + "kind": "TODO", + "content": "this file exists so marker scans have a predictable hit." + } +] diff --git a/fixtures/golden/minimal/symbol-usePermissions.json b/fixtures/golden/minimal/symbol-usePermissions.json new file mode 100644 index 00000000..0a4e9143 --- /dev/null +++ b/fixtures/golden/minimal/symbol-usePermissions.json @@ -0,0 +1,7 @@ +[ + { + "name": "usePermissions", + "kind": "function", + "file_path": "src/usePermissions.ts" + } +] diff --git a/fixtures/golden/scenarios.external.example.json b/fixtures/golden/scenarios.external.example.json new file mode 100644 index 00000000..d84bda28 --- /dev/null +++ b/fixtures/golden/scenarios.external.example.json @@ -0,0 +1,8 @@ +[ + { + "id": "files-count-smoke", + "prompt": "Smoke: indexed corpus has at least one file row", + "sql": "SELECT COUNT(*) AS n FROM files", + "match": { "kind": "minRows", "min": 1 } + } +] diff --git a/fixtures/golden/scenarios.json b/fixtures/golden/scenarios.json new file mode 100644 index 00000000..908a31b9 --- /dev/null +++ b/fixtures/golden/scenarios.json @@ -0,0 +1,32 @@ +[ + { + "id": "files-count", + "prompt": "How many indexed files exist?", + "sql": "SELECT COUNT(*) AS n FROM files" + }, + { + "id": "symbol-usePermissions", + "prompt": "Where is the usePermissions symbol defined?", + "sql": "SELECT name, kind, file_path FROM symbols WHERE name = 'usePermissions'" + }, + { + "id": "index-summary", + "prompt": "Row counts for main tables (same SQL as --recipe index-summary)", + "recipe": "index-summary" + }, + { + "id": "imports-consumer-alias", + "prompt": "Which alias import does consumer.ts use for the API client?", + "sql": "SELECT file_path, source, specifiers, is_type_only FROM imports WHERE file_path = 'src/consumer.ts' AND source = '~/api/client'" + }, + { + "id": "dependencies-from-consumer", + "prompt": "What files does src/consumer.ts depend on (resolved edges)?", + "sql": "SELECT from_path, to_path FROM dependencies WHERE from_path = 'src/consumer.ts' ORDER BY to_path" + }, + { + "id": "markers-notes-todo", + "prompt": "TODO marker in fixture notes markdown", + "sql": "SELECT file_path, line_number, kind, content FROM markers WHERE file_path = 'src/notes.md'" + } +] diff --git a/fixtures/qa/prompts.external.template.md b/fixtures/qa/prompts.external.template.md new file mode 100644 index 00000000..b55829ba --- /dev/null +++ b/fixtures/qa/prompts.external.template.md @@ -0,0 +1,96 @@ +# External index — testing prompts (template) + +Use with **`CODEMAP_ROOT`** / **`CODEMAP_TEST_BENCH`** pointed at the project under test. Copy to **`fixtures/qa/.local.md`**, fill in **Ground truth** after one index, then run the same prompts **with Codemap** (query / skill) and **without** (Read/Grep only) and compare. + +## How to verify + +Run `codemap query` (or `bun src/index.ts query`) with the SQL in each section. Answers should match before you trust free-form chat prose. + +--- + +## 1. Scale & coverage + +**Prompts** + +1. How many source files are in the Codemap index for this repo? How many React components are indexed? +2. Roughly how many `TODO` markers does the index report vs `NOTE`? + +**SQL checks** + +```sql +SELECT COUNT(*) AS files FROM files; +SELECT COUNT(*) AS components FROM components; +SELECT kind, COUNT(*) AS n FROM markers GROUP BY kind ORDER BY n DESC; +``` + +--- + +## 2. Definition / navigation + +**Prompts** + +1. In which file is the symbol `` defined, and what kind is it (`function`, `const`, …)? +2. What npm or workspace packages does `` import directly (first 10 import sources)? + +**SQL checks** + +```sql +SELECT name, kind, file_path FROM symbols WHERE name = '...'; +SELECT DISTINCT source FROM imports WHERE file_path = '...' ORDER BY source LIMIT 15; +``` + +--- + +## 3. Hot spots (fan-out) + +**Prompts** + +1. What are the top 5 files by **outgoing** dependency count (`dependencies` edges where `from_path` is that file)? +2. Why might `` be a refactor risk? + +**SQL checks** + +```sql +SELECT from_path, COUNT(*) AS deps +FROM dependencies GROUP BY from_path ORDER BY deps DESC LIMIT 5; +``` + +--- + +## 4. Shared modules (alias imports) + +**Prompts** + +1. Which `~/…` import paths are most common? Pick one and estimate how many import statements reference it. +2. Where is `~/api/client/types.gen` (or your top alias) consumed from — routers vs components? + +**SQL checks** + +```sql +SELECT source, COUNT(*) AS c FROM imports WHERE source LIKE '~/%' +GROUP BY source ORDER BY c DESC LIMIT 15; +``` + +--- + +## 5. Clarity vs hallucination + +**Prompts** + +1. Does this repo contain a **generated** route tree file? Name it and say what it likely depends on. +2. List **only** facts you can support from the index: file path + one column value from a query. No guessing. + +**Cross-check** +Open the cited file in the editor and confirm the symbol/import exists on the claimed line if the tool gave line numbers. + +--- + +## 6. A/B protocol (manual) + +| Step | With Codemap | Without | +| ---- | --------------------------------------------- | ------------------------------------- | +| 1 | Answer prompts 1–4 using `query` / skill SQL | Same prompts using Glob + Read + Grep | +| 2 | Paste chat export | Paste chat export | +| 3 | Score: factual errors, hedging, path accuracy | Same | + +Record wall-clock time and token usage if your client shows it. diff --git a/package.json b/package.json index 34155780..751c04b4 100644 --- a/package.json +++ b/package.json @@ -45,7 +45,7 @@ "benchmark": "bun src/benchmark.ts", "build": "tsdown", "changeset": "changeset", - "check": "bun run build && bun run --parallel format:check lint:ci test typecheck", + "check": "bun run build && bun run --parallel format:check lint:ci test typecheck && bun run test:golden", "check-updates": "bun update -i --latest", "clean": "git clean -xdf -e .env -e .codemap.db -e .codemap.db-wal -e .codemap.db-shm", "dev": "bun src/index.ts", @@ -59,10 +59,13 @@ "pack": "bun run build && npm pack", "prepare": "husky || true", "prepublishOnly": "bun run build", + "qa:external": "bun scripts/qa-external-repo.ts", "release": "changeset publish", "test": "bun test", "test:ci": "bun run test:coverage", "test:coverage": "bun test --coverage", + "test:golden": "bun scripts/query-golden.ts", + "test:golden:external": "bun scripts/query-golden.ts --corpus external", "typecheck": "tsgo --noEmit" }, "dependencies": { diff --git a/scripts/qa-external-repo.ts b/scripts/qa-external-repo.ts new file mode 100644 index 00000000..78d2c1d6 --- /dev/null +++ b/scripts/qa-external-repo.ts @@ -0,0 +1,216 @@ +#!/usr/bin/env bun +import { existsSync, readFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { createCodemap } from "../src/api"; + +const CODEMAP_REPO = join(dirname(fileURLToPath(import.meta.url)), ".."); + +function parsePositiveInt(flag: string, raw: string | undefined): number { + if (raw === undefined || raw.startsWith("-")) { + throw new Error(`${flag} requires a positive integer`); + } + const n = Number(raw); + if (!Number.isInteger(n) || n < 1) { + throw new Error(`${flag} must be a positive integer`); + } + return n; +} + +function parseArgs(argv: string[]) { + let root: string | undefined; + let skipBenchmark = false; + let verboseBenchmark = false; + let help = false; + let maxFiles = 200; + let maxSymbols = 25; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === "--help" || a === "-h") help = true; + else if (a === "--skip-benchmark") skipBenchmark = true; + else if (a === "--verbose") verboseBenchmark = true; + else if (a === "--root" && argv[i + 1]) root = resolve(argv[++i]); + else if (a === "--max-files" && argv[i + 1]) { + maxFiles = parsePositiveInt("--max-files", argv[++i]); + } else if (a === "--max-symbols" && argv[i + 1]) { + maxSymbols = parsePositiveInt("--max-symbols", argv[++i]); + } else if (a.startsWith("-")) throw new Error(`Unknown option: ${a}`); + } + return { root, skipBenchmark, verboseBenchmark, help, maxFiles, maxSymbols }; +} + +function resolveExternalRoot(cliRoot: string | undefined): string { + const raw = + cliRoot ?? process.env.CODEMAP_ROOT ?? process.env.CODEMAP_TEST_BENCH; + if (raw === undefined || raw === "") { + console.error( + "Set CODEMAP_ROOT or CODEMAP_TEST_BENCH to the project root (absolute path), or pass --root.\n" + + "Example: CODEMAP_ROOT=/path/to/app bun run qa:external\n" + + "Copy .env.example to .env in this repo and set CODEMAP_TEST_BENCH for day-to-day.\n", + ); + process.exit(1); + } + return resolve(raw); +} + +const args = parseArgs(process.argv.slice(2)); + +if (args.help) { + console.log(`Usage: CODEMAP_ROOT=DIR bun scripts/qa-external-repo.ts [options] + bun scripts/qa-external-repo.ts --root DIR [options] + + Root: --root overrides CODEMAP_ROOT and CODEMAP_TEST_BENCH (same precedence as the CLI). + + --root DIR Optional; overrides env when you do not want .env + --skip-benchmark Only run index + sanity checks (no SQL vs glob benchmark) + --verbose Pass --verbose to src/benchmark.ts + --max-files N Cap indexed file paths to check on disk (default 200) + --max-symbols N Cap symbol line checks (default 25) + -h, --help +`); + process.exit(0); +} + +const root = resolveExternalRoot(args.root); +process.env.CODEMAP_ROOT = root; + +async function main(): Promise { + console.log(`\n === codemap qa:external ===\n root: ${root}\n`); + + const cm = await createCodemap({ root }); + const indexResult = await cm.index({ mode: "full", quiet: true }); + console.log( + ` Index: full rebuild in ${indexResult.elapsedMs.toFixed(0)}ms (indexed ${indexResult.indexed}, skipped ${indexResult.skipped})\n`, + ); + + let failed = 0; + + // --- Files: paths in DB exist on disk --- + const fileRows = cm.query("SELECT path FROM files ORDER BY path") as { + path: string; + }[]; + const totalFiles = fileRows.length; + const take = Math.min(args.maxFiles, totalFiles); + const step = + totalFiles <= take ? 1 : Math.max(1, Math.floor(totalFiles / take)); + let checked = 0; + for (let i = 0; i < totalFiles && checked < take; i += step) { + const fp = fileRows[i]!.path; + const abs = join(root, fp); + if (!existsSync(abs)) { + console.error(` FAIL files: missing on disk: ${fp}`); + failed++; + } + checked++; + } + console.log( + ` Files on disk: checked ${checked}/${totalFiles} sampled paths (step ${step}) — ${failed === 0 ? "ok" : `${failed} missing`}`, + ); + + // --- Symbols: declaration line contains the symbol name --- + const symRows = cm.query( + `SELECT name, file_path, line_start FROM symbols + WHERE line_start IS NOT NULL AND length(name) >= 2 + ORDER BY file_path, line_start + LIMIT ${Math.max(1, args.maxSymbols * 4)}`, + ) as { name: string; file_path: string; line_start: number }[]; + + const generic = new Set([ + "default", + "constructor", + "length", + "name", + "toString", + "valueOf", + ]); + let symChecked = 0; + for (const row of symRows) { + if (symChecked >= args.maxSymbols) break; + if (generic.has(row.name)) continue; + symChecked++; + const abs = join(root, row.file_path); + if (!existsSync(abs)) { + console.error( + ` FAIL symbols: file missing for symbol ${row.name}: ${row.file_path}`, + ); + failed++; + continue; + } + const text = readFileSync(abs, "utf-8"); + const lines = text.split(/\r?\n/); + const line = lines[row.line_start - 1]; + if (line === undefined) { + console.error( + ` FAIL symbols: line ${row.line_start} out of range in ${row.file_path}`, + ); + failed++; + continue; + } + if (!line.includes(row.name)) { + console.error( + ` FAIL symbols: line ${row.line_start} in ${row.file_path} does not contain "${row.name}"`, + ); + console.error(` ${line.slice(0, 200)}`); + failed++; + } + } + console.log( + ` Symbol lines: checked ${symChecked} rows — ${failed === 0 ? "ok" : "see errors above"}`, + ); + + // --- Informative samples (manual cross-check with chat / Read tool) --- + console.log( + "\n --- Sample queries (compare with repo by hand or agent) ---\n", + ); + const fanOut = cm.query( + `SELECT from_path, COUNT(*) AS deps FROM dependencies GROUP BY from_path ORDER BY deps DESC LIMIT 5`, + ); + console.log(" Top dependency fan-out (first 5):"); + console.log(JSON.stringify(fanOut, null, 2)); + const comps = cm.query( + `SELECT name, file_path FROM components ORDER BY name LIMIT 8`, + ); + if (Array.isArray(comps) && comps.length > 0) { + console.log("\n Components (up to 8):"); + console.log(JSON.stringify(comps, null, 2)); + } else { + console.log( + "\n (no React components in index — expected if no JSX in corpus)", + ); + } + + if (args.skipBenchmark) { + console.log( + `\n Skipped benchmark (--skip-benchmark).\n === qa:external done (exit ${failed > 0 ? 1 : 0}) ===\n`, + ); + process.exit(failed > 0 ? 1 : 0); + return; + } + + console.log("\n --- Running src/benchmark.ts (same CODEMAP_ROOT) ---\n"); + const benchArgs = ["src/benchmark.ts"]; + if (args.verboseBenchmark) benchArgs.push("--verbose"); + const proc = Bun.spawn(["bun", ...benchArgs], { + cwd: CODEMAP_REPO, + env: { ...process.env, CODEMAP_ROOT: root }, + stdout: "inherit", + stderr: "inherit", + }); + const code = await proc.exited; + if (code !== 0) { + console.error(`\n benchmark.ts exited with ${code}`); + process.exit(code ?? 1); + } + + console.log( + `\n === qa:external done — structural checks: ${failed === 0 ? "pass" : "FAIL"} ===\n` + + ` Next: run the same prompts with/without Codemap in chat and paste exports for diff review.\n`, + ); + process.exit(failed > 0 ? 1 : 0); +} + +main().catch((e) => { + console.error(e); + process.exit(1); +}); diff --git a/scripts/query-golden.ts b/scripts/query-golden.ts new file mode 100644 index 00000000..11b90a1c --- /dev/null +++ b/scripts/query-golden.ts @@ -0,0 +1,266 @@ +#!/usr/bin/env bun +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { createCodemap } from "../src/api"; +import { getQueryRecipeSql } from "../src/cli/query-recipes"; +import { + type GoldenMatch, + type GoldenScenario, + parseScenariosJson, +} from "./query-golden/schema"; + +const REPO_ROOT = join(dirname(fileURLToPath(import.meta.url)), ".."); + +function parseArgs(argv: string[]) { + let update = false; + let help = false; + let strictBudget = false; + let corpus: "minimal" | "external" = "minimal"; + let root: string | undefined; + let scenariosPath: string | undefined; + let goldenDir: string | undefined; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === "--update") update = true; + else if (a === "--help" || a === "-h") help = true; + else if (a === "--strict-budget") strictBudget = true; + else if (a === "--corpus" && argv[i + 1]) { + const v = argv[++i]; + if (v !== "minimal" && v !== "external") { + throw new Error(`--corpus must be minimal or external, got "${v}"`); + } + corpus = v; + } else if (a === "--root" && argv[i + 1]) root = resolve(argv[++i]); + else if (a === "--scenarios" && argv[i + 1]) { + scenariosPath = resolve(argv[++i]); + } else if (a === "--golden-dir" && argv[i + 1]) { + goldenDir = resolve(argv[++i]); + } else if (a.startsWith("-")) throw new Error(`Unknown option: ${a}`); + } + return { update, help, strictBudget, corpus, root, scenariosPath, goldenDir }; +} + +const argv = parseArgs(process.argv.slice(2)); +const UPDATE = argv.update; +const HELP = argv.help; +const STRICT_BUDGET = argv.strictBudget; + +if (HELP) { + console.log(`Usage: bun scripts/query-golden.ts [options] + +Corpus: + --corpus minimal (default) fixtures/minimal + fixtures/golden/scenarios.json + --corpus external Index CODEMAP_ROOT or --root; scenarios from scenarios.external.json + if present, else scenarios.external.example.json; goldens in + fixtures/golden/external/ (gitignored — for local / private trees) + +Options: + --root DIR Project root for --corpus external (else CODEMAP_ROOT / CODEMAP_TEST_BENCH) + --scenarios FILE Override scenarios JSON path + --golden-dir DIR Override golden JSON directory + --update Rewrite golden files from current indexer output + --strict-budget Exit 1 if any scenario exceeds budgetMs (default: warn only) + --help, -h +`); + process.exit(0); +} + +function stableStringify(value: unknown): string { + if (value === null || typeof value !== "object") { + return JSON.stringify(value); + } + if (Array.isArray(value)) { + return `[${value.map((x) => stableStringify(x)).join(",")}]`; + } + const o = value as Record; + const keys = Object.keys(o).sort(); + return `{${keys.map((k) => `${JSON.stringify(k)}:${stableStringify(o[k])}`).join(",")}}`; +} + +function resolveSql(s: GoldenScenario): string { + if (s.sql !== undefined) return s.sql; + if (s.recipe !== undefined) { + const sql = getQueryRecipeSql(s.recipe); + if (sql === undefined) { + throw new Error(`Scenario "${s.id}": unknown recipe "${s.recipe}"`); + } + return sql; + } + throw new Error(`Scenario "${s.id}": missing sql or recipe`); +} + +function defaultMatch(s: GoldenScenario): GoldenMatch { + return s.match ?? { kind: "exact" }; +} + +function evaluateMatch( + rows: unknown[], + match: GoldenMatch, +): { ok: boolean; detail: string } { + if (match.kind === "exact") { + return { ok: true, detail: "" }; + } + if (match.kind === "minRows") { + const ok = rows.length >= match.min; + return { + ok, + detail: ok + ? "" + : `minRows: expected >= ${match.min} rows, got ${rows.length}`, + }; + } + if (match.kind === "everyRowContains") { + for (let i = 0; i < rows.length; i++) { + const r = rows[i]; + if (r === null || typeof r !== "object") { + return { + ok: false, + detail: `everyRowContains: row ${i} is not an object`, + }; + } + const o = r as Record; + const v = o[match.field]; + if (typeof v !== "string" || !v.includes(match.includes)) { + return { + ok: false, + detail: `everyRowContains: row ${i} field ${JSON.stringify(match.field)} must include ${JSON.stringify(match.includes)}`, + }; + } + } + return { ok: true, detail: "" }; + } + return { ok: false, detail: "unknown match kind" }; +} + +async function main(): Promise { + const envRoot = process.env.CODEMAP_ROOT ?? process.env.CODEMAP_TEST_BENCH; + + let fixtureRoot: string; + let scenariosFile: string; + let goldenDir: string; + + if (argv.corpus === "minimal") { + fixtureRoot = join(REPO_ROOT, "fixtures/minimal"); + scenariosFile = + argv.scenariosPath ?? join(REPO_ROOT, "fixtures/golden/scenarios.json"); + goldenDir = argv.goldenDir ?? join(REPO_ROOT, "fixtures/golden/minimal"); + } else { + const rootArg = argv.root ?? (envRoot ? resolve(envRoot) : undefined); + if (rootArg === undefined) { + throw new Error( + "--corpus external requires --root or CODEMAP_ROOT / CODEMAP_TEST_BENCH", + ); + } + fixtureRoot = rootArg; + scenariosFile = + argv.scenariosPath ?? + (existsSync(join(REPO_ROOT, "fixtures/golden/scenarios.external.json")) + ? join(REPO_ROOT, "fixtures/golden/scenarios.external.json") + : join(REPO_ROOT, "fixtures/golden/scenarios.external.example.json")); + goldenDir = argv.goldenDir ?? join(REPO_ROOT, "fixtures/golden/external"); + } + + const raw = readFileSync(scenariosFile, "utf-8"); + const scenarios = parseScenariosJson(raw); + + mkdirSync(goldenDir, { recursive: true }); + + const cm = await createCodemap({ root: fixtureRoot }); + await cm.index({ mode: "full", quiet: true }); + + const modeLabel = UPDATE ? "--update" : "compare"; + const corpusLabel = argv.corpus; + console.log(`\n === query-golden ${modeLabel} (${corpusLabel}) ===`); + if (UPDATE) { + console.log(` (rewriting ${goldenDir}/*.json)\n`); + } else { + console.log(` (${fixtureRoot} indexed vs ${goldenDir}/)\n`); + } + + let failed = 0; + let budgetFailures = 0; + + for (const s of scenarios) { + const sql = resolveSql(s); + const t0 = performance.now(); + const rows = cm.query(sql) as unknown[]; + const durationMs = performance.now() - t0; + const match = defaultMatch(s); + + if (s.budgetMs !== undefined && durationMs > s.budgetMs) { + const msg = ` budget: ${s.id} took ${durationMs.toFixed(1)}ms (limit ${s.budgetMs}ms)`; + if (STRICT_BUDGET) { + console.error(msg); + budgetFailures++; + } else { + console.warn(msg); + } + } + + const goldenPath = join(goldenDir, `${s.id}.json`); + + if (UPDATE) { + writeFileSync(goldenPath, `${JSON.stringify(rows, null, 2)}\n`, "utf-8"); + console.log(` updated ${goldenPath}`); + continue; + } + + if (match.kind === "exact") { + if (!existsSync(goldenPath)) { + console.error(` FAIL: ${s.id} (exact match requires ${goldenPath})`); + failed++; + continue; + } + const expectedRaw = readFileSync(goldenPath, "utf-8"); + const expected = stableStringify(JSON.parse(expectedRaw) as unknown[]); + const actual = stableStringify(rows); + if (actual !== expected) { + console.error(` FAIL: ${s.id}`); + console.error(` expected: ${expected}`); + console.error(` actual: ${actual}`); + failed++; + } else { + console.log(` ok ${s.id}`); + } + continue; + } + + const ev = evaluateMatch(rows, match); + if (!ev.ok) { + console.error(` FAIL: ${s.id}`); + console.error(` ${ev.detail}`); + failed++; + } else { + console.log(` ok ${s.id} (${match.kind})`); + } + } + + if (UPDATE) { + console.log( + "\n Golden files updated. Review diffs before committing.\n === end query-golden --update (exit 0) ===\n", + ); + return; + } + + if (budgetFailures > 0) { + console.error( + `\n query-golden: ${budgetFailures} scenario(s) exceeded budget (--strict-budget).\n`, + ); + process.exit(1); + } + + if (failed > 0) { + console.error(`\n query-golden: ${failed} scenario(s) failed.\n`); + process.exit(1); + } + console.log( + `\n query-golden: all scenarios passed.\n === end query-golden compare (exit 0) ===\n`, + ); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/scripts/query-golden/schema.ts b/scripts/query-golden/schema.ts new file mode 100644 index 00000000..a0b69b96 --- /dev/null +++ b/scripts/query-golden/schema.ts @@ -0,0 +1,49 @@ +import { z } from "zod"; + +const matchExactSchema = z.object({ kind: z.literal("exact") }); + +const matchMinRowsSchema = z.object({ + kind: z.literal("minRows"), + min: z.number().int().nonnegative(), +}); + +const matchEveryRowContainsSchema = z.object({ + kind: z.literal("everyRowContains"), + field: z.string(), + includes: z.string(), +}); + +export const matchSchema = z.union([ + matchExactSchema, + matchMinRowsSchema, + matchEveryRowContainsSchema, +]); + +export type GoldenMatch = z.infer; + +export const scenarioSchema = z + .object({ + id: z.string().min(1), + prompt: z.string().optional(), + sql: z.string().optional(), + recipe: z.string().optional(), + match: matchSchema.optional(), + budgetMs: z.number().positive().optional(), + }) + .refine( + (s) => { + const hasSql = typeof s.sql === "string" && s.sql.length > 0; + const hasRecipe = typeof s.recipe === "string" && s.recipe.length > 0; + return hasSql !== hasRecipe; + }, + { message: "Scenario must have exactly one of sql or recipe" }, + ); + +export type GoldenScenario = z.infer; + +export const scenariosFileSchema = z.array(scenarioSchema); + +export function parseScenariosJson(raw: string): GoldenScenario[] { + const data: unknown = JSON.parse(raw); + return scenariosFileSchema.parse(data); +} diff --git a/src/adapters/builtin.ts b/src/adapters/builtin.ts index 90eaf50a..4aa94383 100644 --- a/src/adapters/builtin.ts +++ b/src/adapters/builtin.ts @@ -45,7 +45,9 @@ function parseText(ctx: ParseContext): ParsedFilePayload { }; } -/** Built-in adapters (oxc TS/JS, Lightning CSS, text/markers). Order matters for the first match. */ +/** + * Built-in adapters (oxc TS/JS, Lightning CSS, text/markers). Order matters for the first match. + */ export const BUILTIN_ADAPTERS: readonly LanguageAdapter[] = [ { id: "builtin.ts-js", diff --git a/src/adapters/types.ts b/src/adapters/types.ts index 93215be5..e0f7af38 100644 --- a/src/adapters/types.ts +++ b/src/adapters/types.ts @@ -4,9 +4,13 @@ import type { ParsedFile } from "../parsed-types"; * Input for a {@link LanguageAdapter}. Paths are absolute / project-relative as noted. */ export interface ParseContext { - /** Absolute path on disk (for parsers that need a real `filename`). */ + /** + * Absolute path on disk (for parsers that need a real `filename`). + */ absPath: string; - /** Path relative to project root (stored in DB rows). */ + /** + * Path relative to project root (stored in DB rows). + */ relPath: string; source: string; } @@ -39,7 +43,9 @@ export type ParsedFilePayload = Pick< */ export interface LanguageAdapter { readonly id: string; - /** Extensions with leading dot, e.g. `.ts`, `.tsx`. */ + /** + * Extensions with leading dot, e.g. `.ts`, `.tsx`. + */ readonly extensions: readonly string[]; parse(ctx: ParseContext): ParsedFilePayload; } diff --git a/src/api.ts b/src/api.ts index 9638200b..48d07187 100644 --- a/src/api.ts +++ b/src/api.ts @@ -24,7 +24,9 @@ export type { } from "./application/types"; export type { RunIndexOptions as IndexOptions } from "./application/run-index"; -/** Database handle returned by `openDb()`; use with {@link runCodemapIndex} in advanced scenarios. */ +/** + * Database handle returned by `openDb()`; use with {@link runCodemapIndex} in advanced scenarios. + */ export type { CodemapDatabase } from "./db"; /** @@ -74,12 +76,16 @@ export async function createCodemap( * Each {@link query} opens the database for that call; {@link index} manages its own open/close lifecycle. */ export class Codemap { - /** Absolute project root (from resolved config). */ + /** + * Absolute project root (from resolved config). + */ get root(): string { return getProjectRoot(); } - /** Absolute path to the SQLite index file (e.g. `.codemap.db`). */ + /** + * Absolute path to the SQLite index file (e.g. `.codemap.db`). + */ get databasePath(): string { return getDatabasePath(); } diff --git a/src/application/index-engine.ts b/src/application/index-engine.ts index 8905eab9..9a82ff09 100644 --- a/src/application/index-engine.ts +++ b/src/application/index-engine.ts @@ -77,6 +77,10 @@ export function collectFiles(): string[] { return [...new Set(files)].sort(); } +// Incremental indexing: `last_indexed_commit` must still be an ancestor of HEAD (otherwise +// history was rewritten — caller does a full rebuild). Union `git diff` (committed deltas +// since that commit) with `git status --porcelain` (staged + unstaged not in the diff alone). +// Filter to extensions we index; `stat` splits live files vs deletions. export function getChangedFiles(db: CodemapDatabase): { changed: string[]; deleted: string[]; @@ -114,6 +118,7 @@ export function getChangedFiles(db: CodemapDatabase): { .trim() .split("\n") .filter(Boolean); + // Porcelain lines are `XY path` (two status chars + space); skip the prefix to get the path. const statusFiles = statusResult.stdout .toString() .trim() @@ -170,12 +175,15 @@ function insertParsedResults( if (parsed.category === "text") { if (parsed.markers?.length) insertMarkers(db, parsed.markers); } else if (parsed.category === "css") { - if (parsed.cssVariables?.length) + if (parsed.cssVariables?.length) { insertCssVariables(db, parsed.cssVariables); - if (parsed.cssClasses?.length) + } + if (parsed.cssClasses?.length) { insertCssClasses(db, parsed.cssClasses); - if (parsed.cssKeyframes?.length) + } + if (parsed.cssKeyframes?.length) { insertCssKeyframes(db, parsed.cssKeyframes); + } if (parsed.markers?.length) insertMarkers(db, parsed.markers); if (parsed.cssImportSources) { @@ -203,8 +211,9 @@ function insertParsedResults( } if (parsed.exports?.length) insertExports(db, parsed.exports); - if (parsed.components?.length) + if (parsed.components?.length) { insertComponents(db, parsed.components); + } if (parsed.markers?.length) insertMarkers(db, parsed.markers); } } catch (err) { @@ -316,11 +325,13 @@ export async function indexFiles( if (markers.length) insertMarkers(db, markers); } else if (category === "css") { const cssData = extractCssData(absPath, source, relPath); - if (cssData.variables.length) + if (cssData.variables.length) { insertCssVariables(db, cssData.variables); + } if (cssData.classes.length) insertCssClasses(db, cssData.classes); - if (cssData.keyframes.length) + if (cssData.keyframes.length) { insertCssKeyframes(db, cssData.keyframes); + } if (cssData.markers.length) insertMarkers(db, cssData.markers); for (const importSource of cssData.importSources) { insertImports(db, [ @@ -434,27 +445,43 @@ export async function targetedReindex( } /** - * Run SQL and print results to stdout (`console.table`), or a friendly error to stderr. - * Does not throw on invalid SQL (matches CLI `query` UX). + * Run read-only SQL and print results to stdout (`console.table`, or JSON when `opts.json`). + * Does not throw on invalid SQL: prints an error and returns **1** (CLI-style). With **`json`**, errors are printed as **`{"error":""}`** on stdout. + * @returns **0** on success, **1** on SQL/runtime error. */ -export function printQueryResult(sql: string): void { - const db = openDb(); +export function printQueryResult( + sql: string, + opts?: { json?: boolean }, +): number { + const json = opts?.json === true; + let db: CodemapDatabase | undefined; try { + db = openDb(); const rows = db.query(sql).all(); - if (rows.length === 0) { + if (json) { + console.log(JSON.stringify(rows)); + } else if (rows.length === 0) { console.log("(no results)"); } else { console.table(rows); } + return 0; } catch (err) { - console.error(`Query error: ${err instanceof Error ? err.message : err}`); + const msg = err instanceof Error ? err.message : String(err); + if (json) { + console.log(JSON.stringify({ error: msg })); + } else { + console.error(`Query error: ${msg}`); + } + return 1; } finally { - closeDb(db); + if (db !== undefined) closeDb(db); } } /** - * Open the index, run SQL, return all rows, then close (used by the public `Codemap.query` API). + * Open the index, run SQL, return all rows, then close. Used by the public **`Codemap.query`** method. + * @throws On invalid SQL or database errors (same as `better-sqlite3`-style `.all()`). */ export function queryRows(sql: string): unknown[] { const db = openDb(); diff --git a/src/application/run-index.ts b/src/application/run-index.ts index 048e432c..627d8ad5 100644 --- a/src/application/run-index.ts +++ b/src/application/run-index.ts @@ -36,14 +36,18 @@ function emptyStats(): IndexTableStats { export type IndexMode = "incremental" | "full" | "files"; export interface RunIndexOptions { - /** Defaults to `incremental`. */ + /** + * Defaults to `incremental`. + */ mode?: IndexMode; /** * Paths relative to the project root; used only when `mode === "files"`. * Non-indexable extensions are filtered out. */ files?: string[]; - /** Suppresses progress logs; parse failures may still be printed. Defaults to `false`. */ + /** + * Suppresses progress logs; parse failures may still be printed. Defaults to `false`. + */ quiet?: boolean; } diff --git a/src/application/types.ts b/src/application/types.ts index 19bdde47..f415ea38 100644 --- a/src/application/types.ts +++ b/src/application/types.ts @@ -14,7 +14,9 @@ export interface IndexTableStats extends Record { css_keyframes: number; } -/** Per-run counters; see {@link IndexResult} for the public shape returned from indexing APIs. */ +/** + * Per-run counters; see {@link IndexResult} for the public shape returned from indexing APIs. + */ export interface IndexRunStats { indexed: number; skipped: number; @@ -27,11 +29,17 @@ export interface IndexRunStats { * Outcome of `Codemap#index` or `runCodemapIndex` (CLI and programmatic index runs). */ export interface IndexResult { - /** How the index was updated. */ + /** + * How the index was updated. + */ mode: "full" | "incremental" | "files"; - /** Files written or re-indexed in this run. */ + /** + * Files written or re-indexed in this run. + */ indexed: number; - /** Files skipped (unchanged hash). */ + /** + * Files skipped (unchanged hash). + */ skipped: number; elapsedMs: number; stats: IndexTableStats; diff --git a/src/benchmark-common.ts b/src/benchmark-common.ts new file mode 100644 index 00000000..04d114cc --- /dev/null +++ b/src/benchmark-common.ts @@ -0,0 +1,60 @@ +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { globSync } from "./glob-sync"; +import { getProjectRoot, isPathExcluded } from "./runtime"; + +export function globFiles(patterns: string[], cwd: string): string[] { + const files = new Set(); + for (const pattern of patterns) { + for (const p of globSync(pattern, cwd)) { + files.add(p); + } + } + return [...files]; +} + +export function globFilesFiltered(patterns: string[], cwd: string): string[] { + return globFiles(patterns, cwd).filter((p) => !isPathExcluded(p)); +} + +export function readAll( + paths: string[], + cwd: string, +): { totalBytes: number; contents: Map } { + let totalBytes = 0; + const contents = new Map(); + for (const p of paths) { + try { + const content = readFileSync(join(cwd, p), "utf-8"); + totalBytes += Buffer.byteLength(content); + contents.set(p, content); + } catch {} + } + return { totalBytes, contents }; +} + +export function traditionalFanoutImportLines(): { + results: unknown[]; + filesRead: number; + bytesRead: number; +} { + const cwd = getProjectRoot(); + const files = globFilesFiltered(["**/*.{ts,tsx,js,jsx}"], cwd); + const { totalBytes, contents } = readAll(files, cwd); + const importish = /^\s*(?:import\b|export\s+[^;]*\bfrom\b|require\s*\()/; + const counts = new Map(); + for (const [path, content] of contents) { + let n = 0; + for (const line of content.split("\n")) { + if (importish.test(line)) n++; + } + counts.set(path, n); + } + const results = [...counts.entries()] + .filter(([, deps]) => deps > 0) + .sort((a, b) => b[1] - a[1]) + .slice(0, 10) + .map(([from_path, deps]) => ({ from_path, deps })); + return { results, filesRead: files.length, bytesRead: totalBytes }; +} diff --git a/src/benchmark-config.test.ts b/src/benchmark-config.test.ts new file mode 100644 index 00000000..fb5623ed --- /dev/null +++ b/src/benchmark-config.test.ts @@ -0,0 +1,33 @@ +import { describe, expect, it } from "bun:test"; + +import { assertReadOnlyIndexedSql } from "./benchmark-config"; + +describe("assertReadOnlyIndexedSql", () => { + it("allows SELECT", () => { + expect(() => assertReadOnlyIndexedSql("SELECT 1")).not.toThrow(); + }); + + it("allows WITH … SELECT", () => { + expect(() => + assertReadOnlyIndexedSql("WITH t AS (SELECT 1 AS x) SELECT * FROM t"), + ).not.toThrow(); + }); + + it("rejects multiple statements", () => { + expect(() => assertReadOnlyIndexedSql("SELECT 1; SELECT 2")).toThrow( + /single statement/, + ); + }); + + it("rejects DELETE", () => { + expect(() => assertReadOnlyIndexedSql("DELETE FROM files")).toThrow( + /read-only/, + ); + }); + + it("rejects RETURNING", () => { + expect(() => assertReadOnlyIndexedSql("SELECT 1 RETURNING 2")).toThrow( + /RETURNING/, + ); + }); +}); diff --git a/src/benchmark-config.ts b/src/benchmark-config.ts new file mode 100644 index 00000000..b72a6815 --- /dev/null +++ b/src/benchmark-config.ts @@ -0,0 +1,186 @@ +import { existsSync, readFileSync } from "node:fs"; +import { resolve } from "node:path"; + +import { + globFilesFiltered, + readAll, + traditionalFanoutImportLines, +} from "./benchmark-common"; +import type { Scenario } from "./benchmark-default-scenarios"; +import type { CodemapDatabase } from "./db"; +import { getProjectRoot } from "./runtime"; + +interface TraditionalRegexSpec { + globs: string[]; + regex: string; + mode: "files" | "matches"; +} + +interface TraditionalBuiltinSpec { + builtin: "fanoutImportLines"; +} + +type TraditionalSpec = TraditionalRegexSpec | TraditionalBuiltinSpec; + +interface ConfigScenario { + name: string; + indexedSql: string; + traditional: TraditionalSpec; +} + +interface BenchmarkConfigFile { + /** Always normalized in {@link parseConfigJson} (default **true**). */ + replaceDefault: boolean; + scenarios: ConfigScenario[]; +} + +/** + * Reject mutating or multi-statement SQL. Benchmark JSON is local/trusted but must not run DDL/DML against `.codemap.db`. + */ +export function assertReadOnlyIndexedSql(sql: string): void { + const trimmed = sql.trim(); + if (trimmed === "") throw new Error("indexedSql must be non-empty"); + const oneStmt = trimmed.replace(/;\s*$/u, "").trim(); + if (oneStmt.includes(";")) { + throw new Error("indexedSql must be a single statement"); + } + if ( + /\b(?:INSERT|UPDATE|DELETE|CREATE|ALTER|DROP|REPLACE|ATTACH|DETACH|VACUUM|PRAGMA|TRUNCATE|REINDEX|ANALYZE)\b/i.test( + oneStmt, + ) + ) { + throw new Error( + "indexedSql must be read-only (no DDL/DML or PRAGMA keywords)", + ); + } + if (/\bRETURNING\b/i.test(oneStmt)) { + throw new Error("indexedSql must not use RETURNING"); + } + if (!/^\s*(?:WITH\b|SELECT\b)/iu.test(oneStmt)) { + throw new Error( + "indexedSql must be a single SELECT (optionally WITH … SELECT)", + ); + } +} + +function isBuiltin(t: TraditionalSpec): t is TraditionalBuiltinSpec { + return "builtin" in t && t.builtin === "fanoutImportLines"; +} + +function traditionalFromSpec(spec: TraditionalSpec): () => { + results: unknown[]; + filesRead: number; + bytesRead: number; +} { + if (isBuiltin(spec)) { + return traditionalFanoutImportLines; + } + const { globs, regex, mode } = spec; + if (!globs?.length || !regex) { + throw new Error( + "traditional: need globs + regex, or builtin fanoutImportLines", + ); + } + return () => { + const cwd = getProjectRoot(); + const files = globFilesFiltered(globs, cwd); + const { totalBytes, contents } = readAll(files, cwd); + const results: unknown[] = []; + if (mode === "matches") { + const re = new RegExp(regex, "g"); + for (const [path, content] of contents) { + re.lastIndex = 0; + let m; + while ((m = re.exec(content)) !== null) { + results.push({ file_path: path, match: m[0] }); + } + } + } else { + // `regex` comes from developer-controlled benchmark JSON (trusted input). + const re = new RegExp(regex); + for (const [path, content] of contents) { + if (re.test(content)) results.push({ file_path: path }); + } + } + return { results, filesRead: files.length, bytesRead: totalBytes }; + }; +} + +function parseConfigJson(raw: string): BenchmarkConfigFile { + const data: unknown = JSON.parse(raw); + if (data === null || typeof data !== "object") { + throw new Error("benchmark config: expected object"); + } + const o = data as Record; + const scenarios = o.scenarios; + if (!Array.isArray(scenarios) || scenarios.length === 0) { + throw new Error("benchmark config: scenarios must be a non-empty array"); + } + for (const s of scenarios) { + if (s === null || typeof s !== "object") { + throw new Error("benchmark config: invalid scenario entry"); + } + const e = s as Record; + if (typeof e.name !== "string" || e.name.length === 0) { + throw new Error("benchmark config: each scenario needs a name"); + } + if (typeof e.indexedSql !== "string" || e.indexedSql.trim() === "") { + throw new Error(`benchmark config: ${e.name}: indexedSql required`); + } + try { + assertReadOnlyIndexedSql(e.indexedSql); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + throw new Error(`benchmark config: ${e.name}: ${msg}`); + } + if (e.traditional === null || typeof e.traditional !== "object") { + throw new Error(`benchmark config: ${e.name}: traditional required`); + } + const t = e.traditional as Record; + if (t.builtin === "fanoutImportLines") continue; + if (!Array.isArray(t.globs) || t.globs.some((g) => typeof g !== "string")) { + throw new Error( + `benchmark config: ${e.name}: traditional.globs must be string[]`, + ); + } + if (typeof t.regex !== "string") { + throw new Error( + `benchmark config: ${e.name}: traditional.regex required`, + ); + } + if (t.mode !== "files" && t.mode !== "matches") { + throw new Error( + `benchmark config: ${e.name}: traditional.mode must be "files" or "matches"`, + ); + } + } + return { + replaceDefault: o.replaceDefault !== false, + scenarios: scenarios as ConfigScenario[], + }; +} + +/** + * Load scenarios from a JSON file (**`CODEMAP_BENCHMARK_CONFIG`**). + * The path is resolved from **`process.cwd()`** via **`resolve(configPath)`** (not relative to this module); use an absolute path or a path relative to the shell cwd. + */ +export function loadScenariosFromConfigFile( + db: CodemapDatabase, + configPath: string, +): { replaceDefault: boolean; scenarios: Scenario[] } { + const resolved = resolve(configPath); + if (!existsSync(resolved)) { + throw new Error(`CODEMAP_BENCHMARK_CONFIG: file not found: ${resolved}`); + } + const raw = readFileSync(resolved, "utf-8"); + const config = parseConfigJson(raw); + const scenarios: Scenario[] = config.scenarios.map((s) => ({ + name: s.name, + indexed: () => db.query(s.indexedSql).all(), + traditional: traditionalFromSpec(s.traditional), + })); + return { + replaceDefault: config.replaceDefault, + scenarios, + }; +} diff --git a/src/benchmark-default-scenarios.ts b/src/benchmark-default-scenarios.ts new file mode 100644 index 00000000..a81bcfda --- /dev/null +++ b/src/benchmark-default-scenarios.ts @@ -0,0 +1,191 @@ +import { + globFilesFiltered, + readAll, + traditionalFanoutImportLines, +} from "./benchmark-common"; +import { getQueryRecipeSql } from "./cli/query-recipes"; +import type { CodemapDatabase } from "./db"; +import { getProjectRoot } from "./runtime"; + +export interface Scenario { + name: string; + indexed: () => unknown[]; + traditional: () => { + results: unknown[]; + filesRead: number; + bytesRead: number; + }; +} + +export function getDefaultScenarios(db: CodemapDatabase): Scenario[] { + return [ + { + name: "Find where 'usePermissions' is defined", + indexed: () => + db + .query( + `SELECT file_path, line_start, line_end, signature + FROM symbols WHERE name = 'usePermissions' AND kind IN ('function', 'variable')`, + ) + .all(), + traditional: () => { + const files = globFilesFiltered(["**/*.{ts,tsx}"], getProjectRoot()); + const { totalBytes, contents } = readAll(files, getProjectRoot()); + const re = /export\s+(?:function|const)\s+usePermissions/; + const results = []; + for (const [path, content] of contents) { + if (re.test(content)) results.push({ file_path: path }); + } + return { results, filesRead: files.length, bytesRead: totalBytes }; + }, + }, + + { + name: "List React components (TSX/JSX)", + indexed: () => + db.query(`SELECT name, file_path FROM components ORDER BY name`).all(), + traditional: () => { + const files = globFilesFiltered(["**/*.{tsx,jsx}"], getProjectRoot()); + const { totalBytes, contents } = readAll(files, getProjectRoot()); + const re = /export\s+(?:default\s+)?(?:function|const)\s+([A-Z]\w*)/g; + const results = []; + for (const [path, content] of contents) { + let m; + while ((m = re.exec(content)) !== null) { + results.push({ file_path: path, name: m[1] }); + } + } + return { results, filesRead: files.length, bytesRead: totalBytes }; + }, + }, + + { + name: "Files that import from ~/api/client", + indexed: () => + db + .query( + `SELECT file_path FROM imports + WHERE source LIKE '~/api/client%' + GROUP BY file_path`, + ) + .all(), + traditional: () => { + const files = globFilesFiltered(["**/*.{ts,tsx}"], getProjectRoot()); + const { totalBytes, contents } = readAll(files, getProjectRoot()); + const re = /from\s+['"]~\/api\/client/; + const results = []; + for (const [path, content] of contents) { + if (re.test(content)) results.push({ file_path: path }); + } + return { results, filesRead: files.length, bytesRead: totalBytes }; + }, + }, + + { + name: "Find all TODO/FIXME markers", + indexed: () => + db + .query(`SELECT file_path, line_number, content, kind FROM markers`) + .all(), + traditional: () => { + const files = globFilesFiltered( + ["**/*.{ts,tsx,css,md}"], + getProjectRoot(), + ); + const { totalBytes, contents } = readAll(files, getProjectRoot()); + const re = /\b(TODO|FIXME|HACK|NOTE)[\s:]+(.+)/g; + const results = []; + for (const [path, content] of contents) { + let m; + while ((m = re.exec(content)) !== null) { + results.push({ file_path: path, kind: m[1], text: m[2]?.trim() }); + } + } + return { results, filesRead: files.length, bytesRead: totalBytes }; + }, + }, + + { + name: "CSS design tokens (custom properties)", + indexed: () => + db + .query( + `SELECT name, value, scope, file_path FROM css_variables ORDER BY name LIMIT 50`, + ) + .all(), + traditional: () => { + const files = globFilesFiltered(["**/*.css"], getProjectRoot()); + const { totalBytes, contents } = readAll(files, getProjectRoot()); + const re = /(--[\w-]+)\s*:\s*([^;]+)/g; + const results = []; + for (const [path, content] of contents) { + let m; + while ((m = re.exec(content)) !== null) { + results.push({ file_path: path, name: m[1], value: m[2]?.trim() }); + } + } + return { results, filesRead: files.length, bytesRead: totalBytes }; + }, + }, + + { + name: "Components in `shop/` subtree", + indexed: () => + db + .query( + `SELECT name, file_path FROM components + WHERE file_path LIKE '%/components/shop/%' + AND (file_path LIKE '%.tsx' OR file_path LIKE '%.jsx') + ORDER BY name`, + ) + .all(), + traditional: () => { + const files = globFilesFiltered( + ["**/components/shop/**/*.tsx", "**/components/shop/**/*.jsx"], + getProjectRoot(), + ); + const { totalBytes, contents } = readAll(files, getProjectRoot()); + const re = /export\s+(?:default\s+)?(?:function|const)\s+(\w+)/g; + const results = []; + for (const [path, content] of contents) { + let m; + while ((m = re.exec(content)) !== null) { + results.push({ file_path: path, name: m[1] }); + } + } + return { results, filesRead: files.length, bytesRead: totalBytes }; + }, + }, + + { + name: "Reverse deps: who imports utils/date?", + indexed: () => + db + .query( + `SELECT from_path FROM dependencies + WHERE to_path LIKE '%utils/date%'`, + ) + .all(), + traditional: () => { + const files = globFilesFiltered(["**/*.{ts,tsx}"], getProjectRoot()); + const { totalBytes, contents } = readAll(files, getProjectRoot()); + const re = /from\s+['"].*utils\/date['"]/; + const results = []; + for (const [path, content] of contents) { + if (re.test(content)) results.push({ file_path: path }); + } + return { results, filesRead: files.length, bytesRead: totalBytes }; + }, + }, + + { + name: "Top 10 by dependency fan-out", + indexed: () => { + const sql = getQueryRecipeSql("fan-out"); + if (!sql) throw new Error("missing fan-out recipe"); + return db.query(sql).all(); + }, + traditional: traditionalFanoutImportLines, + }, + ]; +} diff --git a/src/benchmark.ts b/src/benchmark.ts index 27203c09..677a194b 100644 --- a/src/benchmark.ts +++ b/src/benchmark.ts @@ -1,34 +1,37 @@ -/** - * Benchmark: Codebase Index (SQL) vs Traditional File Scanning - * - * Compares two approaches to answering common code-discovery questions: - * 1. Indexed — single SQL query against .codemap.db - * 2. Traditional — Glob + readFileSync + regex (simulates what AI tools do) - * - * Usage: - * bun src/benchmark.ts - * CODEMAP_ROOT=/path/to/repo bun src/benchmark.ts --verbose - */ - -import { existsSync, readFileSync } from "node:fs"; +import { existsSync, statSync } from "node:fs"; import { join, resolve } from "node:path"; +import { loadScenariosFromConfigFile } from "./benchmark-config"; +import { + getDefaultScenarios, + type Scenario, +} from "./benchmark-default-scenarios"; import { loadUserConfig, resolveCodemapConfig } from "./config"; import { closeDb, openDb } from "./db"; -import { globSync } from "./glob-sync"; import { configureResolver } from "./resolver"; -import { - getProjectRoot, - getTsconfigPath, - initCodemap, - isPathExcluded, -} from "./runtime"; +import { getProjectRoot, getTsconfigPath, initCodemap } from "./runtime"; const VERBOSE = process.argv.includes("--verbose"); -const bootstrapRoot = process.env.CODEMAP_ROOT - ? resolve(process.env.CODEMAP_ROOT) - : process.cwd(); +const bootstrapRoot = + process.env.CODEMAP_ROOT !== undefined + ? resolve(process.env.CODEMAP_ROOT) + : process.env.CODEMAP_TEST_BENCH !== undefined + ? resolve(process.env.CODEMAP_TEST_BENCH) + : process.cwd(); + +if ( + process.env.CODEMAP_ROOT !== undefined || + process.env.CODEMAP_TEST_BENCH !== undefined +) { + if (!existsSync(bootstrapRoot) || !statSync(bootstrapRoot).isDirectory()) { + console.error( + `\n CODEMAP_ROOT / CODEMAP_TEST_BENCH is not an existing directory:\n ${bootstrapRoot}\n\n Use the real absolute path to the project (documentation paths like /path/to/repo are placeholders).\n Example: CODEMAP_ROOT=$HOME/your-org/your-app bun src/benchmark.ts\n`, + ); + process.exit(1); + } +} + const userConfig = await loadUserConfig(bootstrapRoot); initCodemap(resolveCodemapConfig(bootstrapRoot, userConfig)); configureResolver(getProjectRoot(), getTsconfigPath()); @@ -47,34 +50,6 @@ async function timeMsAsync( return { result, ms: performance.now() - start }; } -function globFiles(patterns: string[], cwd: string): string[] { - const files: string[] = []; - for (const pattern of patterns) { - files.push(...globSync(pattern, cwd)); - } - return files; -} - -function globFilesFiltered(patterns: string[], cwd: string): string[] { - return globFiles(patterns, cwd).filter((p) => !isPathExcluded(p)); -} - -function readAll( - paths: string[], - cwd: string, -): { totalBytes: number; contents: Map } { - let totalBytes = 0; - const contents = new Map(); - for (const p of paths) { - try { - const content = readFileSync(join(cwd, p), "utf-8"); - totalBytes += Buffer.byteLength(content); - contents.set(p, content); - } catch {} - } - return { totalBytes, contents }; -} - function fmtBytes(b: number): string { if (b < 1024) return `${b} B`; if (b < 1024 * 1024) return `${(b / 1024).toFixed(1)} KB`; @@ -85,178 +60,6 @@ function fmtMs(ms: number): string { return ms < 1 ? `${(ms * 1000).toFixed(0)}µs` : `${ms.toFixed(2)}ms`; } -interface Scenario { - name: string; - indexed: () => unknown[]; - traditional: () => { - results: unknown[]; - filesRead: number; - bytesRead: number; - }; -} - -const db = openDb(); - -const scenarios: Scenario[] = [ - { - name: "Find where 'usePermissions' is defined", - indexed: () => - db - .query( - `SELECT file_path, line_start, line_end, signature - FROM symbols WHERE name = 'usePermissions' AND kind IN ('function', 'variable')`, - ) - .all(), - traditional: () => { - const files = globFilesFiltered(["**/*.{ts,tsx}"], getProjectRoot()); - const { totalBytes, contents } = readAll(files, getProjectRoot()); - const re = /export\s+(?:function|const)\s+usePermissions/; - const results = []; - for (const [path, content] of contents) { - if (re.test(content)) results.push({ file_path: path }); - } - return { results, filesRead: files.length, bytesRead: totalBytes }; - }, - }, - - { - name: "List React components (TSX/JSX)", - indexed: () => - db.query(`SELECT name, file_path FROM components ORDER BY name`).all(), - traditional: () => { - const files = globFilesFiltered(["**/*.{tsx,jsx}"], getProjectRoot()); - const { totalBytes, contents } = readAll(files, getProjectRoot()); - const re = /export\s+(?:default\s+)?(?:function|const)\s+([A-Z]\w*)/g; - const results = []; - for (const [path, content] of contents) { - let m; - while ((m = re.exec(content)) !== null) { - results.push({ file_path: path, name: m[1] }); - } - } - return { results, filesRead: files.length, bytesRead: totalBytes }; - }, - }, - - { - name: "Files that import from ~/api/client", - indexed: () => - db - .query( - `SELECT file_path FROM imports - WHERE source LIKE '~/api/client%' - GROUP BY file_path`, - ) - .all(), - traditional: () => { - const files = globFilesFiltered(["**/*.{ts,tsx}"], getProjectRoot()); - const { totalBytes, contents } = readAll(files, getProjectRoot()); - const re = /from\s+['"]~\/api\/client/; - const results = []; - for (const [path, content] of contents) { - if (re.test(content)) results.push({ file_path: path }); - } - return { results, filesRead: files.length, bytesRead: totalBytes }; - }, - }, - - { - name: "Find all TODO/FIXME markers", - indexed: () => - db - .query(`SELECT file_path, line_number, content, kind FROM markers`) - .all(), - traditional: () => { - const files = globFilesFiltered( - ["**/*.{ts,tsx,css,md}"], - getProjectRoot(), - ); - const { totalBytes, contents } = readAll(files, getProjectRoot()); - const re = /\b(TODO|FIXME|HACK|NOTE)[\s:]+(.+)/g; - const results = []; - for (const [path, content] of contents) { - let m; - while ((m = re.exec(content)) !== null) { - results.push({ file_path: path, kind: m[1], text: m[2]?.trim() }); - } - } - return { results, filesRead: files.length, bytesRead: totalBytes }; - }, - }, - - { - name: "CSS design tokens (custom properties)", - indexed: () => - db - .query( - `SELECT name, value, scope, file_path FROM css_variables ORDER BY name LIMIT 50`, - ) - .all(), - traditional: () => { - const files = globFilesFiltered(["**/*.css"], getProjectRoot()); - const { totalBytes, contents } = readAll(files, getProjectRoot()); - const re = /(--[\w-]+)\s*:\s*([^;]+)/g; - const results = []; - for (const [path, content] of contents) { - let m; - while ((m = re.exec(content)) !== null) { - results.push({ file_path: path, name: m[1], value: m[2]?.trim() }); - } - } - return { results, filesRead: files.length, bytesRead: totalBytes }; - }, - }, - - { - name: "Components in `shop/` subtree", - indexed: () => - db - .query( - `SELECT name, file_path FROM components - WHERE file_path LIKE '%/components/%shop%' - ORDER BY name`, - ) - .all(), - traditional: () => { - const files = globFilesFiltered( - ["**/components/shop/**/*.tsx"], - getProjectRoot(), - ); - const { totalBytes, contents } = readAll(files, getProjectRoot()); - const re = /export\s+(?:default\s+)?(?:function|const)\s+(\w+)/g; - const results = []; - for (const [path, content] of contents) { - let m; - while ((m = re.exec(content)) !== null) { - results.push({ file_path: path, name: m[1] }); - } - } - return { results, filesRead: files.length, bytesRead: totalBytes }; - }, - }, - - { - name: "Reverse deps: who imports utils/date?", - indexed: () => - db - .query( - `SELECT from_path FROM dependencies - WHERE to_path LIKE '%utils/date%'`, - ) - .all(), - traditional: () => { - const files = globFilesFiltered(["**/*.{ts,tsx}"], getProjectRoot()); - const { totalBytes, contents } = readAll(files, getProjectRoot()); - const re = /from\s+['"].*utils\/date['"]/; - const results = []; - for (const [path, content] of contents) { - if (re.test(content)) results.push({ file_path: path }); - } - return { results, filesRead: files.length, bytesRead: totalBytes }; - }, - }, -]; - interface Row { scenario: string; indexedMs: string; @@ -271,10 +74,28 @@ interface Row { speedup: string; } -// Warmup query to prime SQLite page cache -db.query("SELECT COUNT(*) FROM files").get(); +const db = openDb(); -console.log("\n Codemap — Benchmark\n"); +const configPath = process.env.CODEMAP_BENCHMARK_CONFIG; +let scenarios: Scenario[]; +if (configPath !== undefined && configPath !== "") { + const resolvedConfig = resolve(configPath); + const loaded = loadScenariosFromConfigFile(db, configPath); + if (loaded.replaceDefault) { + scenarios = loaded.scenarios; + } else { + scenarios = [...getDefaultScenarios(db), ...loaded.scenarios]; + } + const mergeNote = loaded.replaceDefault ? "" : " + defaults"; + console.log( + `\n Codemap — Benchmark (${scenarios.length} scenario(s)${mergeNote} from ${resolvedConfig})\n`, + ); +} else { + scenarios = getDefaultScenarios(db); + console.log("\n Codemap — Benchmark\n"); +} + +db.query("SELECT COUNT(*) FROM files").get(); const rows: Row[] = []; @@ -347,7 +168,7 @@ console.log( `\n Totals: Index ${fmtMs(totalIdxMs)} vs Traditional ${fmtMs(totalTradMs)} (${(totalTradMs / Math.max(totalIdxMs, 0.001)).toFixed(1)}× overall)\n`, ); -const avgTokensPerByte = 0.25; // ~4 bytes per token (rough) +const avgTokensPerByte = 0.25; const totalBytesTraditional = rows.reduce((s, r) => s + r.bytesReadRaw, 0); const estimatedTokens = Math.round(totalBytesTraditional * avgTokensPerByte); console.log(` Token impact estimate:`); diff --git a/src/cli.test.ts b/src/cli.test.ts index bfc53e42..96869bc9 100644 --- a/src/cli.test.ts +++ b/src/cli.test.ts @@ -35,6 +35,23 @@ describe("CLI --help", () => { expect(out).toContain("codemap query"); expect(err).toBe(""); }); + + test("query --help exits 0 and documents --json", async () => { + const { exitCode, out, err } = await runCli(["query", "--help"]); + expect(exitCode).toBe(0); + expect(out).toContain("--json"); + expect(out).toContain("--recipe"); + expect(out).toContain("fan-out"); + expect(out).toContain("codemap query"); + expect(out).toContain("LIMIT"); + expect(err).toBe(""); + }); + + test("query with no SQL exits 1", async () => { + const { exitCode, err } = await runCli(["query"]); + expect(exitCode).toBe(1); + expect(err).toContain("missing SQL"); + }); }); describe("CLI version", () => { diff --git a/src/cli/bootstrap.ts b/src/cli/bootstrap.ts index c45b26e0..dab44498 100644 --- a/src/cli/bootstrap.ts +++ b/src/cli/bootstrap.ts @@ -2,7 +2,9 @@ import { resolve } from "node:path"; import { CODEMAP_VERSION } from "../version"; -/** Printed for `codemap --help` / `-h` (must run before config or DB access). */ +/** + * Printed for `codemap --help` / `-h` (must run before config or DB access). + */ export function printCliUsage(): void { console.log(`Usage: codemap [options] [command] @@ -11,7 +13,8 @@ Index (default): update .codemap.db for the project root (\`--root\` or cwd). codemap [--root DIR] [--config FILE] --files Query: - codemap query "" + codemap query [--json] "" + codemap query [--json] --recipe Agents: codemap agents init [--force] [--interactive|-i] diff --git a/src/cli/cmd-query.test.ts b/src/cli/cmd-query.test.ts new file mode 100644 index 00000000..deffdd97 --- /dev/null +++ b/src/cli/cmd-query.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, it } from "bun:test"; + +import { parseQueryRest } from "./cmd-query"; +import { getQueryRecipeSql, listQueryRecipeCatalog } from "./query-recipes"; + +describe("parseQueryRest", () => { + it("errors when only query", () => { + const r = parseQueryRest(["query"]); + expect(r.kind).toBe("error"); + if (r.kind === "error") expect(r.message).toContain("missing SQL"); + }); + + it("returns help for query --help", () => { + expect(parseQueryRest(["query", "--help"]).kind).toBe("help"); + expect(parseQueryRest(["query", "-h"]).kind).toBe("help"); + }); + + it("parses SQL after query", () => { + const r = parseQueryRest(["query", "SELECT", "1"]); + expect(r).toEqual({ kind: "run", sql: "SELECT 1", json: false }); + }); + + it("parses --json and SQL", () => { + const r = parseQueryRest(["query", "--json", "SELECT", "1"]); + expect(r).toEqual({ kind: "run", sql: "SELECT 1", json: true }); + }); + + it("errors when --json has no SQL", () => { + const r = parseQueryRest(["query", "--json"]); + expect(r.kind).toBe("error"); + if (r.kind === "error") { + expect(r.message).toContain("missing SQL or recipe"); + } + }); + + it("returns help for query --json --help", () => { + expect(parseQueryRest(["query", "--json", "--help"]).kind).toBe("help"); + }); + + it("parses --recipe fan-out-sample-json", () => { + const r = parseQueryRest(["query", "--recipe", "fan-out-sample-json"]); + const sql = getQueryRecipeSql("fan-out-sample-json"); + expect(sql).toBeDefined(); + expect(r).toEqual({ + kind: "run", + sql: sql!, + json: false, + }); + }); + + it("parses --recipe fan-out", () => { + const r = parseQueryRest(["query", "--recipe", "fan-out"]); + const sql = getQueryRecipeSql("fan-out"); + expect(sql).toBeDefined(); + expect(r).toEqual({ + kind: "run", + sql: sql!, + json: false, + }); + }); + + it("parses --json --recipe fan-out-sample", () => { + const r = parseQueryRest(["query", "--json", "--recipe", "fan-out-sample"]); + const sql = getQueryRecipeSql("fan-out-sample"); + expect(sql).toBeDefined(); + expect(r).toEqual({ + kind: "run", + sql: sql!, + json: true, + }); + }); + + it("parses --recipe fan-out --json", () => { + const r = parseQueryRest(["query", "--recipe", "fan-out", "--json"]); + const sql = getQueryRecipeSql("fan-out"); + expect(sql).toBeDefined(); + expect(r).toEqual({ + kind: "run", + sql: sql!, + json: true, + }); + }); + + it("errors on unknown recipe", () => { + const r = parseQueryRest(["query", "--recipe", "nope"]); + expect(r.kind).toBe("error"); + if (r.kind === "error") { + expect(r.message).toContain("unknown recipe"); + expect(r.message).toContain("fan-out"); + } + }); + + it("errors when --recipe has no id", () => { + const r = parseQueryRest(["query", "--recipe"]); + expect(r.kind).toBe("error"); + if (r.kind === "error") expect(r.message).toContain("--recipe"); + }); + + it("errors when extra tokens after recipe", () => { + const r = parseQueryRest(["query", "--recipe", "fan-out", "SELECT", "1"]); + expect(r.kind).toBe("error"); + if (r.kind === "error") expect(r.message).toContain("does not take"); + }); + + it("parses --recipes-json", () => { + expect(parseQueryRest(["query", "--recipes-json"])).toEqual({ + kind: "recipesCatalog", + }); + }); + + it("parses --json --recipes-json", () => { + expect(parseQueryRest(["query", "--json", "--recipes-json"])).toEqual({ + kind: "recipesCatalog", + }); + }); + + it("errors when --recipes-json has extra args", () => { + const r = parseQueryRest(["query", "--recipes-json", "SELECT", "1"]); + expect(r.kind).toBe("error"); + if (r.kind === "error") expect(r.message).toContain("--recipes-json"); + }); + + it("errors when --recipes-json combines with --recipe", () => { + const r = parseQueryRest([ + "query", + "--recipes-json", + "--recipe", + "fan-out", + ]); + expect(r.kind).toBe("error"); + if (r.kind === "error") expect(r.message).toContain("--recipe"); + }); + + it("parses --print-sql fan-out", () => { + expect(parseQueryRest(["query", "--print-sql", "fan-out"])).toEqual({ + kind: "printRecipeSql", + id: "fan-out", + }); + }); + + it("errors when --print-sql has unknown id", () => { + const r = parseQueryRest(["query", "--print-sql", "nope"]); + expect(r.kind).toBe("error"); + if (r.kind === "error") expect(r.message).toContain("unknown recipe"); + }); + + it("errors when --print-sql has no id", () => { + const r = parseQueryRest(["query", "--print-sql"]); + expect(r.kind).toBe("error"); + if (r.kind === "error") expect(r.message).toContain("--print-sql"); + }); +}); + +describe("listQueryRecipeCatalog", () => { + it("matches QUERY_RECIPES ids and sql", () => { + const cat = listQueryRecipeCatalog(); + expect(cat.length).toBeGreaterThan(0); + for (const row of cat) { + expect(getQueryRecipeSql(row.id)).toBe(row.sql); + expect(row.description.length).toBeGreaterThan(0); + } + }); +}); diff --git a/src/cli/cmd-query.ts b/src/cli/cmd-query.ts index 78a78d7a..441e03c5 100644 --- a/src/cli/cmd-query.ts +++ b/src/cli/cmd-query.ts @@ -2,14 +2,242 @@ import { printQueryResult } from "../application/index-engine"; import { loadUserConfig, resolveCodemapConfig } from "../config"; import { configureResolver } from "../resolver"; import { getProjectRoot, getTsconfigPath, initCodemap } from "../runtime"; +import { + getQueryRecipeSql, + listQueryRecipeCatalog, + listQueryRecipeIds, + QUERY_RECIPES, +} from "./query-recipes"; +/** + * Parse `argv` after the global bootstrap: `rest[0]` must be `"query"`. + * Supports `--json`, `--recipe `, `--recipes-json`, `--print-sql `, and raw SQL (see {@link printQueryCmdHelp}). + */ +export function parseQueryRest( + rest: string[], +): + | { kind: "help" } + | { kind: "error"; message: string } + | { kind: "run"; sql: string; json: boolean } + | { kind: "recipesCatalog" } + | { kind: "printRecipeSql"; id: string } { + if (rest[0] !== "query") { + throw new Error("parseQueryRest: expected query"); + } + if (rest.length === 1) { + return { + kind: "error", + message: + 'codemap: missing SQL or recipe. Usage: codemap query [--json] "" | codemap query [--json] --recipe | codemap query --recipes-json | codemap query --print-sql \nRun codemap query --help for more.', + }; + } + + let i = 1; + let json = false; + let recipeId: string | undefined; + let recipesJson = false; + let printSqlId: string | undefined; + + while (i < rest.length) { + const a = rest[i]; + if (a === "--help" || a === "-h") { + return { kind: "help" }; + } + if (a === "--json") { + json = true; + i++; + continue; + } + if (a === "--recipes-json") { + recipesJson = true; + i++; + continue; + } + if (a === "--print-sql") { + const name = rest[i + 1]; + if (name === undefined || name.startsWith("-")) { + return { + kind: "error", + message: + 'codemap: "--print-sql" requires a recipe id. Example: codemap query --print-sql fan-out', + }; + } + printSqlId = name; + i += 2; + continue; + } + if (a === "--recipe") { + const name = rest[i + 1]; + if (name === undefined || name.startsWith("-")) { + return { + kind: "error", + message: + 'codemap: "--recipe" requires a recipe id. Example: codemap query --recipe fan-out', + }; + } + recipeId = name; + i += 2; + continue; + } + break; + } + + if (recipesJson) { + if (recipeId !== undefined || printSqlId !== undefined) { + return { + kind: "error", + message: + "codemap: --recipes-json cannot be combined with --recipe or --print-sql.", + }; + } + if (i < rest.length) { + return { + kind: "error", + message: + "codemap: --recipes-json does not take SQL or extra arguments.", + }; + } + return { kind: "recipesCatalog" }; + } + + if (printSqlId !== undefined) { + if (recipeId !== undefined) { + return { + kind: "error", + message: "codemap: use either --recipe or --print-sql, not both.", + }; + } + if (i < rest.length) { + return { + kind: "error", + message: + "codemap: --print-sql does not take a SQL string; only the recipe id.", + }; + } + const sql = getQueryRecipeSql(printSqlId); + if (sql === undefined) { + const known = listQueryRecipeIds().join(", "); + return { + kind: "error", + message: `codemap: unknown recipe "${printSqlId}". Known recipes: ${known}`, + }; + } + return { kind: "printRecipeSql", id: printSqlId }; + } + + if (recipeId !== undefined) { + if (i < rest.length) { + return { + kind: "error", + message: + "codemap: --recipe does not take a SQL string; remove arguments after the recipe id.", + }; + } + const sql = getQueryRecipeSql(recipeId); + if (sql === undefined) { + const known = listQueryRecipeIds().join(", "); + return { + kind: "error", + message: `codemap: unknown recipe "${recipeId}". Known recipes: ${known}`, + }; + } + return { kind: "run", sql, json }; + } + + const sql = rest.slice(i).join(" ").trim(); + if (!sql) { + return { + kind: "error", + message: + 'codemap: missing SQL or recipe. Usage: codemap query [--json] "" | codemap query [--json] --recipe | codemap query --recipes-json | codemap query --print-sql ', + }; + } + return { kind: "run", sql, json }; +} + +/** Print the bundled recipe catalog as JSON to stdout (no DB access). */ +export function printRecipesCatalogJson(): void { + console.log(JSON.stringify(listQueryRecipeCatalog(), null, 2)); +} + +/** Print one recipe's SQL to stdout, or false if the id is unknown (caller should exit 1). */ +export function printRecipeSqlToStdout(id: string): boolean { + const sql = getQueryRecipeSql(id); + if (sql === undefined) { + return false; + } + console.log(sql); + return true; +} + +function formatRecipeHelpLines(): string { + const ids = listQueryRecipeIds(); + const lines = ids.map((id) => { + const meta = QUERY_RECIPES[id]; + const desc = meta?.description ?? ""; + return ` ${id.padEnd(16)} ${desc}`; + }); + return lines.join("\n"); +} + +/** + * Print **`codemap query`** usage, flags, and bundled recipe ids to stdout. + */ +export function printQueryCmdHelp(): void { + const recipeBlock = formatRecipeHelpLines(); + console.log(`Usage: codemap query [--json] "" + codemap query [--json] --recipe + codemap query --recipes-json + codemap query --print-sql + +Read-only SQL against .codemap.db (after at least one successful index run). +The CLI does not cap row count — use SQL LIMIT (and ORDER BY) when you need a bounded result set. + + --json Print a JSON array of row objects to stdout (for agents and scripts). + On error, prints a single object: {"error":""} to stdout. + + --recipe Run bundled SQL (no SQL string on the command line). + + --recipes-json Print all bundled recipes (id, description, sql) as JSON to stdout. No DB. + + --print-sql Print one recipe's SQL text to stdout (does not run the query). No DB. + +Bundled recipes: +${recipeBlock} + +Examples: + codemap query "SELECT name, file_path FROM symbols LIMIT 10" + codemap query --json "SELECT COUNT(*) AS n FROM symbols" + codemap query --recipe fan-out + codemap query --json --recipe fan-out-sample + codemap query --recipes-json + codemap query --print-sql fan-out +`); +} + +/** + * Initialize Codemap for `opts.root`, then run **`printQueryResult`**. + * Sets **`process.exitCode`** on failure (no **`process.exit`**). With **`--json`**, bootstrap errors print **`{"error":"…"}`** on stdout like query failures. + */ export async function runQueryCmd(opts: { root: string; configFile: string | undefined; sql: string; + json?: boolean; }): Promise { - const user = await loadUserConfig(opts.root, opts.configFile); - initCodemap(resolveCodemapConfig(opts.root, user)); - configureResolver(getProjectRoot(), getTsconfigPath()); - printQueryResult(opts.sql); + try { + const user = await loadUserConfig(opts.root, opts.configFile); + initCodemap(resolveCodemapConfig(opts.root, user)); + configureResolver(getProjectRoot(), getTsconfigPath()); + const code = printQueryResult(opts.sql, { json: opts.json }); + if (code !== 0) process.exitCode = code; + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + if (opts.json) { + console.log(JSON.stringify({ error: msg })); + } else { + console.error(msg); + } + process.exitCode = 1; + } } diff --git a/src/cli/main.ts b/src/cli/main.ts index af090e45..aaec4068 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -66,20 +66,38 @@ Copies bundled agent templates into .agents/ under the project root. validateIndexModeArgs(rest); - if (rest[0] === "query" && rest[1]) { - if (rest[1] === "--help" || rest[1] === "-h") { - console.log(`Usage: codemap query "" - -Runs read-only SQL against .codemap.db (after at least one successful index run). -Example: codemap query "SELECT name, file_path FROM symbols LIMIT 10" -`); + if (rest[0] === "query") { + const { + parseQueryRest, + printQueryCmdHelp, + printRecipesCatalogJson, + printRecipeSqlToStdout, + runQueryCmd, + } = await import("./cmd-query.js"); + const parsed = parseQueryRest(rest); + if (parsed.kind === "help") { + printQueryCmdHelp(); + return; + } + if (parsed.kind === "error") { + console.error(parsed.message); + process.exit(1); + } + if (parsed.kind === "recipesCatalog") { + printRecipesCatalogJson(); + return; + } + if (parsed.kind === "printRecipeSql") { + if (!printRecipeSqlToStdout(parsed.id)) { + process.exit(1); + } return; } - const { runQueryCmd } = await import("./cmd-query.js"); await runQueryCmd({ root, configFile, - sql: rest.slice(1).join(" "), + sql: parsed.sql, + json: parsed.json, }); return; } diff --git a/src/cli/query-recipes.ts b/src/cli/query-recipes.ts new file mode 100644 index 00000000..770cfa52 --- /dev/null +++ b/src/cli/query-recipes.ts @@ -0,0 +1,131 @@ +/** + * One bundled recipe: id, human description, and SQL (canonical source for CLI and `--recipes-json`). + */ +export type QueryRecipeCatalogEntry = { + id: string; + description: string; + sql: string; +}; + +/** + * Bundled read-only SQL for `codemap query --recipe `. Keys match **`codemap query --help`**. + */ +export const QUERY_RECIPES: Record< + string, + { sql: string; description: string } +> = { + "fan-out": { + description: "Top 10 files by dependency fan-out (edge count)", + sql: `SELECT from_path, COUNT(*) AS deps +FROM dependencies +GROUP BY from_path +ORDER BY deps DESC, from_path ASC +LIMIT 10`, + }, + "fan-out-sample": { + description: + "Top 10 by fan-out, plus up to five sample dependency targets per file", + sql: `SELECT d.from_path, + COUNT(*) AS deps, + (SELECT GROUP_CONCAT(to_path, ' | ') + FROM (SELECT to_path FROM dependencies d2 WHERE d2.from_path = d.from_path ORDER BY to_path ASC LIMIT 5)) + AS sample_targets +FROM dependencies d +GROUP BY d.from_path +ORDER BY deps DESC, d.from_path ASC +LIMIT 10`, + }, + /** + * Same ranking as `fan-out-sample`, but sample targets as a JSON array (SQLite JSON1 + * `json_group_array`). Prefer `fan-out-sample` if JSON1 is unavailable. + */ + "fan-out-sample-json": { + description: + "Like fan-out-sample, but sample_targets is a JSON array (requires JSON1)", + sql: `SELECT d.from_path, + COUNT(*) AS deps, + (SELECT json_group_array(to_path) + FROM (SELECT to_path FROM dependencies d2 WHERE d2.from_path = d.from_path ORDER BY to_path ASC LIMIT 5)) + AS sample_targets +FROM dependencies d +GROUP BY d.from_path +ORDER BY deps DESC, d.from_path ASC +LIMIT 10`, + }, + /** + * Files most imported/depended-on (complement to fan-out). + */ + "fan-in": { + description: "Top 15 files by fan-in (how many other files depend on them)", + sql: `SELECT to_path, COUNT(*) AS fan_in +FROM dependencies +GROUP BY to_path +ORDER BY fan_in DESC, to_path ASC +LIMIT 15`, + }, + "index-summary": { + description: + "Single row: row counts for main tables (quick health snapshot)", + sql: `SELECT + (SELECT COUNT(*) FROM files) AS files, + (SELECT COUNT(*) FROM symbols) AS symbols, + (SELECT COUNT(*) FROM imports) AS imports, + (SELECT COUNT(*) FROM components) AS components, + (SELECT COUNT(*) FROM dependencies) AS dependencies`, + }, + "files-largest": { + description: "Top 20 files by line count (size/complexity hotspots)", + sql: `SELECT path, line_count, size, language +FROM files +ORDER BY line_count DESC, path ASC +LIMIT 20`, + }, + /** + * Hook count uses comma tally + 1 on the stored JSON array (Codemap emits flat + * `["useFoo","useBar"]` shapes). Avoids SQLite JSON1 (`json_array_length`) so + * the recipe runs on any SQLite build the CLI already supports. + */ + "components-by-hooks": { + description: + "React components with the most hooks (comma count on stored JSON array)", + sql: `SELECT name, file_path, + CASE + WHEN hooks_used IS NULL OR trim(hooks_used) = '' OR trim(hooks_used) = '[]' THEN 0 + ELSE (length(hooks_used) - length(replace(hooks_used, ',', ''))) + 1 + END AS hook_count +FROM components +ORDER BY hook_count DESC, file_path ASC, name ASC +LIMIT 20`, + }, + "markers-by-kind": { + description: "Marker counts by kind (TODO, FIXME, …)", + sql: `SELECT kind, COUNT(*) AS count +FROM markers +GROUP BY kind +ORDER BY count DESC, kind ASC`, + }, +}; + +/** + * Sorted recipe ids (same set as {@link QUERY_RECIPES}). + */ +export function listQueryRecipeIds(): string[] { + return Object.keys(QUERY_RECIPES).sort(); +} + +/** + * Full catalog for **`codemap query --recipes-json`** — derived from {@link QUERY_RECIPES} only. + */ +export function listQueryRecipeCatalog(): QueryRecipeCatalogEntry[] { + return listQueryRecipeIds().map((id) => { + const meta = QUERY_RECIPES[id]!; + return { id, description: meta.description, sql: meta.sql }; + }); +} + +/** + * Returns the SQL string for a recipe id, or `undefined` if unknown. + */ +export function getQueryRecipeSql(id: string): string | undefined { + return QUERY_RECIPES[id]?.sql; +} diff --git a/src/config.ts b/src/config.ts index 1375ef8a..a38b95f8 100644 --- a/src/config.ts +++ b/src/config.ts @@ -81,7 +81,9 @@ export const codemapUserConfigSchema = z }) .strict(); -/** Inferred from {@link codemapUserConfigSchema}. */ +/** + * Inferred from {@link codemapUserConfigSchema}. + */ export type CodemapUserConfig = z.infer; function formatCodemapConfigError(error: z.ZodError): string { diff --git a/src/css-parser.ts b/src/css-parser.ts index 1466517e..6c776d53 100644 --- a/src/css-parser.ts +++ b/src/css-parser.ts @@ -133,8 +133,9 @@ function stringifyCssValue(value: any): string { function stringifyToken(token: any): string { if (!token) return ""; if (typeof token === "string") return token; - if (token.type === "length") + if (token.type === "length") { return `${token.value?.value ?? 0}${token.value?.unit ?? ""}`; + } if (token.type === "percentage") return `${token.value ?? 0}%`; if (token.type === "color") return stringifyColor(token.value); if (token.type === "token") { @@ -154,8 +155,9 @@ function stringifyToken(token: any): string { function stringifyColor(color: any): string { if (!color) return ""; if (color.type === "rgb") return `rgb(${color.r}, ${color.g}, ${color.b})`; - if (color.type === "rgba") + if (color.type === "rgba") { return `rgba(${color.r}, ${color.g}, ${color.b}, ${color.alpha})`; + } return JSON.stringify(color); } diff --git a/src/version.ts b/src/version.ts index a6228bb4..401ccf96 100644 --- a/src/version.ts +++ b/src/version.ts @@ -1,4 +1,6 @@ import packageJson from "../package.json" with { type: "json" }; -/** Package version from `package.json` (inlined at build time). */ +/** + * Package version from `package.json` (inlined at build time). + */ export const CODEMAP_VERSION = packageJson.version; diff --git a/templates/agents/rules/codemap.mdc b/templates/agents/rules/codemap.mdc index ad995686..34e7130f 100644 --- a/templates/agents/rules/codemap.mdc +++ b/templates/agents/rules/codemap.mdc @@ -22,6 +22,10 @@ Install **[@stainless-code/codemap](https://www.npmjs.com/package/@stainless-cod | ------ | ------- | | Incremental index | `codemap` | | Query | `codemap query ""` | +| Query (JSON) | `codemap query --json ""` | +| Query (recipe) | `codemap query --recipe fan-out` (see **`codemap query --help`**) | +| Recipe catalog (JSON) | `codemap query --recipes-json` | +| Print one recipe’s SQL | `codemap query --print-sql fan-out` | **Bundled rules/skills:** **`codemap agents init`** writes **`.agents/`** from the package (see [docs/agents.md](../../../docs/agents.md)). @@ -74,6 +78,10 @@ If the question looks like any of these → use the index: codemap query "" ``` +**Row count:** The CLI does **not** impose a maximum number of rows. Add **`LIMIT`** (and **`ORDER BY`**) in SQL when you need a bounded list. For automation and multi-row answers, prefer **`codemap query --json`** — stdout is a JSON array; on failure, stdout is **`{"error":""}`** and the process exits **1**. + +**Verbatim answers:** When the user asks for lists, counts, or enumerated structural data from the index, **paste or summarize from the query output without inventing rows** — do not substitute a prose “summary” that omits rows the user asked to see. Prefer **`--json`** so the full result set is unambiguous. + ## Quick reference queries | I need to... | Query | diff --git a/templates/agents/skills/codemap/SKILL.md b/templates/agents/skills/codemap/SKILL.md index 026eb528..a47b74d2 100644 --- a/templates/agents/skills/codemap/SKILL.md +++ b/templates/agents/skills/codemap/SKILL.md @@ -16,6 +16,44 @@ codemap query "" Use **`codemap --root /path/to/project`** (or **`CODEMAP_ROOT`**) to index another tree. +## Query output and agents + +- **`codemap query --json`** prints a **JSON array** of row objects to stdout. On SQL error, stdout is **`{"error":""}`** and the process exits **1**. +- The CLI **does not cap** how many rows SQLite returns — add **`LIMIT`** and **`ORDER BY`** in SQL when you need a bounded list. +- When answering with structural facts from the index (lists of paths, symbols, dependency edges), **ground the answer in the query rows** — do not invent or silently drop rows. Prefer **`--json`** for large or multi-column results. + +## Agent-friendly SQL recipes + +Replace placeholders (`'...'`) with your module path, file glob, or symbol name. + +**CLI shortcuts:** **`codemap query --recipe `** runs bundled SQL (optional **`--json`**). **`codemap query --recipes-json`** prints every bundled recipe (**`id`**, **`description`**, **`sql`**) as JSON (no index / DB required). **`codemap query --print-sql `** prints one recipe’s SQL only. Ids include **`fan-out`**, **`fan-out-sample`** (**`GROUP_CONCAT`** samples), **`fan-out-sample-json`** (same, but **`json_group_array`** — needs SQLite JSON1), **`fan-in`**, **`index-summary`**, **`files-largest`**, **`components-by-hooks`**, **`markers-by-kind`** — see **`codemap query --help`**. The fan-out rows match the SQL below; others align with “Conditional aggregation”, “Codebase statistics”, and component sections later in this skill. + +**Top files by dependency fan-out:** + +```sql +SELECT from_path, COUNT(*) AS deps +FROM dependencies +GROUP BY from_path +ORDER BY deps DESC +LIMIT 10; +``` + +**Same ranking, plus up to five sample targets per file** (uses a correlated subquery; adjust **`LIMIT 5`** as needed): + +```sql +SELECT d.from_path, + COUNT(*) AS deps, + (SELECT GROUP_CONCAT(to_path, ' | ') + FROM (SELECT to_path FROM dependencies d2 WHERE d2.from_path = d.from_path LIMIT 5)) + AS sample_targets +FROM dependencies d +GROUP BY d.from_path +ORDER BY deps DESC +LIMIT 10; +``` + +**JSON array samples (JSON1):** replace **`GROUP_CONCAT`** with **`json_group_array(to_path)`** in that subquery if your SQLite build has JSON1 — or use **`codemap query --recipe fan-out-sample-json`**. + ## Schema ### `files` — Every indexed file @@ -196,6 +234,8 @@ SELECT name, file_path, hooks_used FROM components WHERE hooks_used LIKE '%useTheme%'; -- Components with most hooks (complexity indicator) +-- `json_array_length` requires SQLite JSON1. For a portable ranking, use +-- `codemap query --recipe components-by-hooks` (comma-based count on the stored JSON array). SELECT name, file_path, json_array_length(hooks_used) as hook_count FROM components ORDER BY hook_count DESC LIMIT 15; diff --git a/tsconfig.json b/tsconfig.json index a096fee8..1c4a4456 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -12,5 +12,11 @@ "noUnusedLocals": true, "noUnusedParameters": true }, - "include": ["src/**/*.ts", "tsdown.config.ts"] + "include": [ + "src/**/*.ts", + "tsdown.config.ts", + "scripts/query-golden.ts", + "scripts/query-golden/**/*.ts", + "scripts/qa-external-repo.ts" + ] }