|
| 1 | +# Plan — `codemap audit --base <ref>` |
| 2 | + |
| 3 | +> Two-snapshot structural-drift verdict for a PR / branch. Adopted from [`docs/research/fallow.md` § Tier B B.5](../research/fallow.md) — explicitly the "single highest-leverage candidate" of that scan. |
| 4 | +
|
| 5 | +**Status:** Open — design pass; not yet implemented. |
| 6 | +**Cross-refs:** [`docs/research/fallow.md`](../research/fallow.md) (motivation) · [`docs/architecture.md` § CLI usage](../architecture.md#cli-usage) (where wiring lands) · [`.agents/lessons.md`](../../.agents/lessons.md) (changesets bump policy). |
| 7 | + |
| 8 | +--- |
| 9 | + |
| 10 | +## 1. Goal |
| 11 | + |
| 12 | +One command returns a structured verdict for what changed between a base ref and `HEAD`: |
| 13 | + |
| 14 | +```text |
| 15 | +codemap audit --base origin/main [--json] [--summary] |
| 16 | +↓ |
| 17 | +{ |
| 18 | + "verdict": "pass" | "warn" | "fail", |
| 19 | + "base": { "ref": "origin/main", "sha": "<sha>", "indexed_at": <ms> }, |
| 20 | + "head": { "sha": "<sha>", "indexed_at": <ms> }, |
| 21 | + "deltas": { |
| 22 | + "files": { "added": [...], "removed": [...] }, |
| 23 | + "dependencies": { "added": [...], "removed": [...] }, |
| 24 | + "deprecated": { "added": [...], "removed": [...] }, |
| 25 | + "visibility": { "added": [...], "removed": [...] }, |
| 26 | + "barrels": { "movements": [...] }, |
| 27 | + "hot_files": { "movements": [...] } |
| 28 | + } |
| 29 | +} |
| 30 | +``` |
| 31 | + |
| 32 | +Wraps existing recipes; doesn't grow a new analysis layer. Stays consistent with codemap's structural-index thesis ([`docs/why-codemap.md` § What Codemap is not](../why-codemap.md#what-codemap-is-not)). |
| 33 | + |
| 34 | +## 2. Non-goals (v1) |
| 35 | + |
| 36 | +- **Dead-code / duplication / complexity verdicts.** Those are fallow's territory and a non-goal per [`docs/roadmap.md` § Non-goals (v1)](../roadmap.md#non-goals-v1). |
| 37 | +- **Code-quality scoring / grading.** No "code health 87/100" output. |
| 38 | +- **Auto-fix / SARIF output.** Separate concerns — SARIF is B.8, auto-fix is explicitly out (D.14 in the research note). |
| 39 | +- **Cross-repo audit** (audit `origin/main` of project A from a checkout of project B). Out of scope; reuse `--root` for the simpler "audit a different tree" case. |
| 40 | +- **Continuous mode.** One-shot CLI, same as `codemap query`. |
| 41 | + |
| 42 | +## 3. Snapshot strategy |
| 43 | + |
| 44 | +The verdict is a diff between two indexed snapshots. Three credible architectures: |
| 45 | + |
| 46 | +### Option A: Temp DB on the base ref (worktree-style) |
| 47 | + |
| 48 | +```text |
| 49 | +1. git worktree add /tmp/codemap-audit-<sha> <base-ref> |
| 50 | +2. codemap --root /tmp/codemap-audit-<sha> --full # builds .codemap.db there |
| 51 | +3. Open both DBs, run delta queries cross-DB, emit verdict. |
| 52 | +4. git worktree remove /tmp/codemap-audit-<sha> |
| 53 | +``` |
| 54 | + |
| 55 | +**Pros:** Same code path as a normal index run on the base; no special "snapshot" abstraction; deltas are pure SQL across two attached DBs; reproducible regardless of how `HEAD` evolves. |
| 56 | + |
| 57 | +**Cons:** Spawns a worktree + full reindex per audit (cold cost ~seconds for codemap-sized projects, more for large monorepos). Disk churn under `/tmp`. |
| 58 | + |
| 59 | +### Option B: In-memory base via the existing `query_baselines` table (B.6 reuse) |
| 60 | + |
| 61 | +```text |
| 62 | +1. On main, periodically: for each "tracked" recipe, codemap query --save-baseline -r <id>. |
| 63 | +2. On a PR branch: codemap audit --base <name> diffs the live query results against the saved snapshots. |
| 64 | +``` |
| 65 | + |
| 66 | +**Pros:** Zero new infra — reuses B.6 directly. Snapshots are addressable / nameable. No cold reindex. |
| 67 | + |
| 68 | +**Cons:** Requires baselines to be saved at the right moment (git-hook or CI step). Doesn't capture deltas the user didn't pre-baseline. Doesn't naturally express "deltas in the dependency graph as a whole" — only as far as recipes go. |
| 69 | + |
| 70 | +### Option C: On-demand snapshot table for the audit (hybrid) |
| 71 | + |
| 72 | +```text |
| 73 | +1. codemap audit --base <ref> reads <ref> from git, computes audit-shaped queries against the |
| 74 | + *checked-out* tree at <ref> (using `git show <ref>:<file>` or `git archive` to materialise |
| 75 | + files in memory / a temp dir), populates a tiny in-DB `audit_snapshot` table with just the |
| 76 | + columns needed for the deltas (no full reindex). |
| 77 | +2. Diff in SQL; drop the snapshot table. |
| 78 | +``` |
| 79 | + |
| 80 | +**Pros:** No worktree spawn; no extra infra in main code paths; deltas are scoped to what the audit needs. |
| 81 | + |
| 82 | +**Cons:** Implementing a "mini-indexer" that runs only the queries we need at <ref> is more code than (A) and the abstraction doesn't transfer. |
| 83 | + |
| 84 | +### Recommendation |
| 85 | + |
| 86 | +**Start with Option A** (temp worktree + full index). Reasons: |
| 87 | + |
| 88 | +1. Simplest to implement correctly — no new abstractions; the existing `--full --root /tmp/...` path already works. |
| 89 | +2. Cold cost on codemap (~150 files) is sub-second; on JordanCoin-sized projects (~few thousand files) still under 5s. Acceptable for "run on PR" usage. |
| 90 | +3. Future optimisation: cache `<sha> → /tmp/codemap-audit-<sha>/.codemap.db` so repeated audits on the same base hit the cache. |
| 91 | +4. Doesn't entangle the audit with B.6's user-facing baseline workflow (which has different semantics: user-named, hand-saved). |
| 92 | + |
| 93 | +**Reconsider Option B** if Option A's perf becomes a problem AND audits are happening in tight loops (e.g. file-watch trigger). |
| 94 | + |
| 95 | +## 4. Built-in deltas (v1) |
| 96 | + |
| 97 | +Each delta wraps an existing query / recipe. All structural — no new analysis layer. |
| 98 | + |
| 99 | +| Delta key | What it surfaces | Source | |
| 100 | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | |
| 101 | +| `files` | New / deleted indexed files | `SELECT path FROM files` (set diff) | |
| 102 | +| `dependencies` | New / deleted edges in the file-to-file dependency graph | `SELECT from_path, to_path FROM dependencies` (set diff) | |
| 103 | +| `deprecated` | New / removed `@deprecated` symbols | `--recipe deprecated-symbols` (set diff) | |
| 104 | +| `visibility` | New / removed visibility-tagged symbols (`@internal` / `@beta` / `@alpha` / `@private` — `@public` is the surface itself, not noise) | `SELECT name, kind, visibility, file_path FROM symbols WHERE visibility IS NOT NULL AND visibility != 'public'` (set diff) | |
| 105 | +| `barrels` | Files that crossed an export-count threshold (e.g. <10 → ≥10) | `--recipe barrel-files` (compare top-N membership) | |
| 106 | +| `hot_files` | Files that gained / lost rank in the fan-in or fan-out top-15 | `--recipe fan-in` / `--recipe fan-out` (compare top-N membership) | |
| 107 | + |
| 108 | +**Out of v1** (reconsider once shipped): |
| 109 | + |
| 110 | +- `cycles` — needs cycle detection on the dependency graph; not a recipe today |
| 111 | +- `boundary_crossings` — needs a project-supplied glob list (similar to the future `audit-pr-architecture` skill kit); no canonical source |
| 112 | +- `markers` — TODO/FIXME drift is noisy and project-specific |
| 113 | +- `css_*` deltas — narrow audience; defer |
| 114 | + |
| 115 | +## 5. Verdict shape |
| 116 | + |
| 117 | +`pass | warn | fail` derived from per-delta thresholds. **Defaults exposed but conservative:** |
| 118 | + |
| 119 | +| Delta | Default threshold | |
| 120 | +| ----- | ----------------------------------------------- | |
| 121 | +| any | `pass` (thresholds are opt-in via config in v1) | |
| 122 | + |
| 123 | +In other words: **v1 emits raw deltas only**. The verdict is always `pass` unless the user opts in via `codemap.config.*`. Reasoning: structural deltas don't have a universally-meaningful threshold ("how many new dependency edges is too many?" depends entirely on the project), and the research note explicitly biases toward "first pass exposes raw deltas only and lets the consumer set thresholds." |
| 124 | + |
| 125 | +### Threshold config (v1.x) |
| 126 | + |
| 127 | +Once per-project use surfaces concrete thresholds, fold into `codemap.config.*`: |
| 128 | + |
| 129 | +```ts |
| 130 | +// codemap.config.ts |
| 131 | +export default defineConfig({ |
| 132 | + audit: { |
| 133 | + deltas: { |
| 134 | + dependencies: { added_max: 50, action: "warn" }, |
| 135 | + deprecated: { added_max: 0, action: "fail" }, // any new @deprecated fails |
| 136 | + visibility: { added_max: 5, action: "warn" }, |
| 137 | + }, |
| 138 | + // verdict reduction: highest action wins (fail > warn > pass) |
| 139 | + }, |
| 140 | +}); |
| 141 | +``` |
| 142 | + |
| 143 | +Validated via existing `codemapUserConfigSchema` (Zod) — see [`docs/architecture.md` § User config](../architecture.md#user-config). Schema additions are minor changesets per [`.agents/lessons.md` "changesets bump policy"](../../.agents/lessons.md) (no `.codemap.db` impact). |
| 144 | + |
| 145 | +## 6. Composition with existing flags |
| 146 | + |
| 147 | +| Flag | Behaviour with `audit` | |
| 148 | +| -------------------------------- | ------------------------------------------------------------------------------------------------------------------ | |
| 149 | +| `--json` | Default for the verdict shape; non-JSON falls back to `console.table` per delta + a one-line verdict summary. | |
| 150 | +| `--summary` | Collapses every delta to `{added: N, removed: N}`; verdict + base/head metadata stay. Useful for CI status checks. | |
| 151 | +| `--changed-since` | **Mutex** — `audit` is itself a "changed-since" operation; combining would be confusing. Parser-level error. | |
| 152 | +| `--group-by` | **Mutex** — verdict shape is already structured; bucketing is the consumer's job on the output JSON. | |
| 153 | +| `--save-baseline` / `--baseline` | **Mutex** — different snapshot semantics (B.6 is user-named; audit is base-ref-driven). | |
| 154 | +| `--recipe` | N/A — `audit` isn't a `query` subcommand; it's its own top-level command. | |
| 155 | + |
| 156 | +## 7. CLI surface |
| 157 | + |
| 158 | +```text |
| 159 | +codemap audit --base <ref> [--json] [--summary] [--root <dir>] [--config <file>] |
| 160 | +``` |
| 161 | + |
| 162 | +- `--base <ref>` — required. Any committish (`origin/main`, `HEAD~5`, sha, tag). |
| 163 | +- `--root` / `--config` / `--help` / `-h` — same shape as the rest of the CLI (handled by `bootstrap`). |
| 164 | +- Exit codes: **0** on `pass`, **1** on `warn`, **2** on `fail`. (CI-friendly; mirrors `git diff --exit-code`.) |
| 165 | + |
| 166 | +## 8. Tracer-bullet sequence |
| 167 | + |
| 168 | +Per [`.agents/rules/tracer-bullets`](../../.agents/rules/tracer-bullets.md), commit each slice end-to-end: |
| 169 | + |
| 170 | +1. **CLI scaffold** — `codemap audit --help` works; `--base <ref>` parsed; `runAuditCmd` calls a stub that returns `{verdict: "pass", deltas: {}}`. Smoke + commit. |
| 171 | +2. **Worktree + base index** — Option A spawn-and-index implementation; assert two `.codemap.db` files exist. Commit. |
| 172 | +3. **First delta — `files`** — minimal end-to-end vertical slice: open both DBs, set-diff `path`, emit `{files: {added, removed}}`. Smoke + commit. |
| 173 | +4. **Remaining deltas** — `dependencies`, `deprecated`, `visibility`, `barrels`, `hot_files` — each as a separate commit so individual tests can be reviewed. |
| 174 | +5. **Threshold config** — Zod schema additions + verdict reduction; default `pass` until user opts in. Commit. |
| 175 | +6. **Docs + agents update** — `architecture.md § Audit wiring`, glossary entry, README CLI block, rule + skill across `.agents/` and `templates/agents/` (Rule 10). Commit. |
| 176 | +7. **Changeset** — patch (no schema bump). Commit. |
| 177 | + |
| 178 | +Estimated total: 1–2 days end-to-end across ~7 commits. |
| 179 | + |
| 180 | +## 9. Open questions |
| 181 | + |
| 182 | +- **Should the temp worktree live under `.codemap/audit-<sha>/` (project-local) or `/tmp/codemap-audit-<sha>` (system temp)?** Project-local is gitignorable via the existing `.codemap.*` glob (works only if the dir is named `.codemap.audit-<sha>`); system temp is auto-cleaned but loses the cache benefit across reboots. **Lean: project-local, naming `.codemap.audit-<sha>` so the existing gitignore covers it.** |
| 183 | +- **Should `audit` warn when `<base>` and `HEAD` are identical?** Almost certainly user error (probably wanted `--base origin/main` not `--base HEAD`). Surface a warning, exit 0 with empty deltas. |
| 184 | +- **Should the verdict include `actions` per delta key?** Recipe `actions` (Tier A.1) attach to row sets; an audit delta is a higher-level concept. v1 punts; v1.x can add `audit.actions: { dependencies: "review-coupling-spike" }` if patterns emerge. |
| 185 | +- **Cross-snapshot performance ceiling.** At what project size does Option A become unacceptable (>30s)? Need a benchmark fixture; defer until a real consumer hits the wall. |
| 186 | + |
| 187 | +## 10. References |
| 188 | + |
| 189 | +- Motivation: [`docs/research/fallow.md` § Tier B B.5](../research/fallow.md) ("single highest-leverage candidate"). |
| 190 | +- Snapshot primitive prior art: PR #30 — `query_baselines` table + `--save-baseline` / `--baseline`. |
| 191 | +- Composition: PR #26 — Tier A flags (`--summary` / `--changed-since` / `--group-by` / per-row `actions`). |
| 192 | +- Visibility column prior art: PR #28 — `symbols.visibility` (B.7). |
| 193 | +- CLI conventions: [`docs/architecture.md` § CLI usage](../architecture.md#cli-usage). |
| 194 | +- Doc lifecycle: this file follows the **Plan** type per [`docs/README.md` § Document Lifecycle](../README.md#document-lifecycle) — **delete on ship**, lift the canonical bits into `architecture.md` per Rule 2. |
0 commit comments