Merge pull request #806 from DeusData/distill/560-architecture-overview

fix(mcp): get_architecture overview subset, aspects enum + validation (+ .cbmignore how-to)

Author: DeusData

README.md

@@ -458,6 +458,8 @@ Anything outside this subset (write/`MERGE`/`CALL` clauses, unsupported function
 
 Layered: hardcoded patterns (`.git`, `node_modules`, etc.) → `.gitignore` hierarchy → `.cbmignore` (project-specific, gitignore syntax). Symlinks are always skipped.
 
+See [docs/cbmignore.md](docs/cbmignore.md) for the full `.cbmignore` how-to: syntax, precedence across the ignore layers, and negation semantics.
+
 ## Configuration
 
 ```bash

docs/cbmignore.md

@@ -0,0 +1,121 @@
+# `.cbmignore` — Excluding Files from Indexing
+
+`.cbmignore` is a project-specific ignore file that controls which files the
+indexer sees. It uses gitignore-style syntax and is read from the **root of
+the indexed directory** (`<repo>/.cbmignore`). Nested `.cbmignore` files in
+subdirectories are not read.
+
+It applies at **file discovery time** — the directory walk that selects files
+for parsing. Every indexing path uses the same discovery: the initial
+`index_repository`, manual re-indexing, and background auto-sync. A path
+matched by `.cbmignore` never enters the graph. Changes to `.cbmignore` take
+effect on the next (re-)index.
+
+Unlike `.gitignore`, it has no effect on git itself — it only shapes what the
+indexer sees. Commit it to share indexing excludes with your team, or list it
+in `.gitignore` to keep personal excludes untracked.
+
+To verify it works: directory subtrees skipped during discovery are reported
+in the `index_repository` response under `excluded`
+(`{"dirs": [up to 25 paths], "count": <total>, "truncated": <bool>}`).
+
+## Syntax
+
+One pattern per line. Blank lines are ignored, lines starting with `#` are
+comments, and trailing whitespace is trimmed.
+
+| Feature | Meaning |
+|---|---|
+| `*` | matches any run of characters, except `/` |
+| `?` | matches exactly one character, except `/` |
+| `**` | matches across directory boundaries (`**/name`, `dir/**`, `a/**/b`) |
+| `[abc]`, `[a-z]` | character classes; `[!a-z]` / `[^a-z]` negate the class |
+| trailing `/` | pattern matches **directories only** |
+| `/` anywhere else | anchors the pattern to the repo root |
+| no `/` in pattern | matches the file/directory name at **any depth** |
+| leading `!` | negation — re-includes a previously matched path; the **last matching pattern wins** |
+
+Examples:
+
+```gitignore
+# Generated protobuf output, anywhere in the tree
+*.pb.go
+
+# A specific top-level directory (leading / anchors to the repo root)
+/third_party/
+
+# Any directory named "snapshots", at any depth (trailing / = directories only)
+snapshots/
+
+# Everything under any fixtures directory
+**/fixtures/**
+
+# Anchored glob: generated clients for any single-character API version
+/api/v?/generated/
+
+# Character class: yearly log folders 2020-2029
+/logs/202[0-9]/
+
+# Ignore all YAML, but keep CI configs (negation — last match wins)
+*.yaml
+!ci.yaml
+```
+
+## Precedence
+
+Discovery applies its filters in a fixed order — the first layer that rejects
+a path wins. For directories:
+
+1. **Built-in skip list** — `.git`, `node_modules`, `dist`, `target`,
+   `vendor`, tool caches, etc. (60+ names; the fast/moderate index modes add
+   more, e.g. `docs`, `examples`, `testdata`). Not overridable from any
+   ignore file today.
+2. **Repo `.gitignore`** — `<repo>/.gitignore` merged with
+   `<git-common-dir>/info/exclude` (worktree-aware); later patterns win on
+   conflict. Honored even when the indexed directory is not a git repo root.
+3. **Nested `.gitignore` files** — picked up during the walk and matched
+   relative to their own directory.
+4. **`.cbmignore`** — a positive match skips the path; a negated match can
+   only rescue paths from layer 5.
+5. **Git global excludes** — `core.excludesFile` from `~/.gitconfig` or the
+   XDG git config (default `$XDG_CONFIG_HOME/git/ignore`); consulted only
+   when the project is a git repo with a config.
+
+For files, built-in suffix filters (`.png`, `.o`, `.db`, …; fast modes add
+archives, media, lockfiles, `.min.js`, …) and fast-mode filename/substring
+filters run **before** the ignore files, and a maximum-file-size cap runs
+after them; none of these are overridable from `.cbmignore`. Symlinks are
+always skipped.
+
+## Negation (`!`) — current behavior
+
+- **Within `.cbmignore`**: standard gitignore semantics. Patterns are
+  evaluated top to bottom and the last matching pattern wins, so
+  `!pattern` re-includes something an earlier line excluded.
+- **Parent pruning** (same caveat as git): when a directory is excluded, the
+  walk never descends into it — you cannot re-include a file whose parent
+  directory is excluded. Negate the directory itself if you need its
+  contents.
+- **Across layers**: a `.cbmignore` negation overrides the **git global
+  excludes** layer only. Example: your `~/.config/git/ignore` ignores
+  `*.sql`, but this project's SQL should be indexed — add `!*.sql` to
+  `.cbmignore`. Negation cannot override the built-in skip lists, the repo
+  `.gitignore`/`info/exclude`, nested `.gitignore` files, the built-in
+  suffix/filename filters, or the size cap.
+
+### Planned (not yet implemented)
+
+The negation story is being unified; none of the following works yet:
+
+- `!` in `.cbmignore` will be able to un-skip ordinary built-in skip
+  directories (`obj/`, `dist/`, `target/`, …) so build-output-like
+  directories that actually contain source can be indexed.
+- A small safety core stays non-negatable by design — `.git`,
+  `node_modules`, and worktree-internal directories — because indexing them
+  risks OOM and correctness issues (see issue #489).
+- Auxiliary filesystem walkers will honor the same ignore predicate as
+  discovery, so every code path sees an identical ignore decision
+  (unification tracked in a follow-up issue).
+
+Until these land, the "Precedence" and "Negation — current behavior" sections
+above describe the actual behavior.

src/mcp/mcp.c

@@ -437,11 +437,15 @@ static const tool_def_t TOOLS[] = {
      "representative top_nodes, and the packages/edge_types that bind it) — use these to grasp "
      "the real architectural seams, which often cut across the folder layout. Optional path scopes "
      "analysis to nodes under that directory prefix (file_path).",
+     /* The aspects enum mirrors VALID_ASPECTS (see aspect_is_valid) — update both together. */
      "{\"type\":\"object\",\"properties\":{\"project\":{\"type\":\"string\"},\"path\":{\"type\":"
      "\"string\",\"description\":\"Optional directory prefix to scope architecture (e.g. "
      "apps/hoa)\"},"
-     "\"aspects\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}}},\"required\":[\"project\"]"
-     "}"},
+     "\"aspects\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"enum\":[\"all\","
+     "\"overview\",\"structure\",\"dependencies\",\"routes\",\"languages\",\"packages\","
+     "\"entry_points\",\"hotspots\",\"boundaries\",\"layers\",\"file_tree\",\"clusters\"]},"
+     "\"description\":\"Aspects to include. 'all' = everything; 'overview' = compact summary "
+     "(all except file_tree); omit = all.\"}},\"required\":[\"project\"]}"},
 
     {"search_code", "Search code",
      "Graph-augmented code search. Finds text patterns via grep, then enriches results with "
@@ -2244,7 +2248,29 @@ static char *handle_delete_project(cbm_mcp_server_t *srv, const char *args) {
     return result;
 }
 
-/* Check if an aspect is requested (NULL aspects = all, or array contains "all" or the name). */
+/* Canonical list of valid aspect tokens for get_architecture. Single source
+ * of truth for the server-side validation (authoritative); the JSON-Schema
+ * enum in the TOOLS entry above is the advisory client-side mirror — update
+ * both together when the aspect set changes. */
+static const char *VALID_ASPECTS[] = {
+    "all",          "overview", "structure",  "dependencies", "routes",    "languages", "packages",
+    "entry_points", "hotspots", "boundaries", "layers",       "file_tree", "clusters",  NULL};
+
+static bool aspect_is_valid(const char *name) {
+    if (!name) {
+        return false;
+    }
+    for (int i = 0; VALID_ASPECTS[i]; i++) {
+        if (strcmp(name, VALID_ASPECTS[i]) == 0) {
+            return true;
+        }
+    }
+    return false;
+}
+
+/* Check if an aspect is requested. NULL aspects = all. The array can contain
+ * "all" (everything), "overview" (everything except file_tree — see
+ * cbm_store_arch_aspect_in_overview in store.c), or the aspect name itself. */
 static bool aspect_wanted(yyjson_doc *aspects_doc, yyjson_val *aspects_arr, const char *name) {
     if (!aspects_arr) {
         return true; /* no filter = all */
@@ -2254,7 +2280,16 @@ static bool aspect_wanted(yyjson_doc *aspects_doc, yyjson_val *aspects_arr, cons
     yyjson_val *val;
     while ((val = yyjson_arr_iter_next(&iter)) != NULL) {
         const char *s = yyjson_get_str(val);
-        if (s && (strcmp(s, "all") == 0 || strcmp(s, name) == 0)) {
+        if (!s) {
+            continue;
+        }
+        if (strcmp(s, "all") == 0) {
+            return true;
+        }
+        if (strcmp(s, "overview") == 0 && cbm_store_arch_aspect_in_overview(name)) {
+            return true;
+        }
+        if (strcmp(s, name) == 0) {
             return true;
         }
     }
@@ -2331,6 +2366,35 @@ static char *handle_get_architecture(cbm_mcp_server_t *srv, const char *args) {
         }
     }
 
+    /* Server-side validation: reject unknown aspect tokens with an isError
+     * result listing the valid values. The JSON-Schema enum is advisory —
+     * many MCP clients do not validate arguments against tool schemas — so
+     * without this check a typo degraded to a silent near-empty payload. */
+    for (int i = 0; i < aspects_strs_count; i++) {
+        if (!aspect_is_valid(aspects_strs[i])) {
+            char valid_list[CBM_SZ_256];
+            size_t off = 0;
+            for (int j = 0; VALID_ASPECTS[j] && off < sizeof(valid_list); j++) {
+                int n = snprintf(valid_list + off, sizeof(valid_list) - off, "%s%s",
+                                 j > 0 ? ", " : "", VALID_ASPECTS[j]);
+                if (n < 0) {
+                    break;
+                }
+                off += (size_t)n;
+            }
+            char msg[CBM_SZ_512];
+            snprintf(msg, sizeof(msg), "Unknown aspect '%s'. Valid: %s.", aspects_strs[i],
+                     valid_list);
+            char *err = cbm_mcp_text_result(msg, true);
+            free(project);
+            free(scope_path);
+            if (aspects_doc) {
+                yyjson_doc_free(aspects_doc);
+            }
+            return err;
+        }
+    }
+
     cbm_schema_info_t schema = {0};
     /* Counts-only: this handler renders label/type counts but never property
      * keys, and full key discovery json_each-scans every row (seconds-to-

src/store/store.c

@@ -5482,6 +5482,15 @@ static int arch_clusters(cbm_store_t *s, const char *project, const char *path,
 
 /* ── GetArchitecture dispatch ──────────────────────────────────── */
 
+/* "overview" = compact architecture summary: every aspect EXCEPT the large
+ * per-file listing (file_tree), which alone dominates the payload on real
+ * repos and can push the MCP response past the output cap. Declared in
+ * store.h and shared with aspect_wanted in src/mcp/mcp.c so the store-side
+ * DB gate and the MCP-side serialization gate cannot drift. */
+bool cbm_store_arch_aspect_in_overview(const char *name) {
+    return strcmp(name, "file_tree") != 0;
+}
+
 static bool want_aspect(const char **aspects, int aspect_count, const char *name) {
     if (!aspects || aspect_count == 0) {
         return true;
@@ -5490,6 +5499,9 @@ static bool want_aspect(const char **aspects, int aspect_count, const char *name
         if (strcmp(aspects[i], "all") == 0) {
             return true;
         }
+        if (strcmp(aspects[i], "overview") == 0 && cbm_store_arch_aspect_in_overview(name)) {
+            return true;
+        }
         if (strcmp(aspects[i], name) == 0) {
             return true;
         }

src/store/store.h

@@ -333,6 +333,12 @@ bool cbm_store_arch_path_scoped(const char *path);
 /* When scoped, writes normalized directory prefix into norm_out. Returns false if unscoped. */
 bool cbm_store_normalize_arch_path(const char *path, char *norm_out, size_t norm_sz);
 
+/* True when architecture aspect `name` belongs to the "overview" subset:
+ * every aspect EXCEPT the large per-file listing (file_tree). Shared by both
+ * aspect gates — want_aspect (store.c) and aspect_wanted (mcp.c) — so the
+ * two sites cannot drift. */
+bool cbm_store_arch_aspect_in_overview(const char *name);
+
 /* Delete all nodes for a project (cascade deletes edges). */
 int cbm_store_delete_nodes_by_project(cbm_store_t *s, const char *project);