|
| 1 | +--- |
| 2 | +title: Multi-lingual, multi-project analysis at scale |
| 3 | +description: Analyze many projects in many languages once, push the results into a shared Neo4j graph, and let agents query all of it through one read-only analysis API. |
| 4 | +--- |
| 5 | + |
| 6 | +import { Tabs, TabItem, Steps, Aside, LinkCard, CardGrid, Badge } from "@astrojs/starlight/components"; |
| 7 | + |
| 8 | +In the [default workflow](/quickstart/), CLDK runs the analysis in-process: you point `CLDK.java(...)` at a project, the backend parses it, and the typed models live in memory for the lifetime of that object. That is the right model for a single project on a single machine. |
| 9 | + |
| 10 | +It does not fit a fleet. When you have hundreds of repositories across several languages, and agents that need to answer structural questions about any of them at any time, re-analyzing on every request is wasteful, and holding every project in memory is impossible. |
| 11 | + |
| 12 | +CLDK supports a second model for exactly this case. **Analysis and querying are split into two phases that scale independently:** |
| 13 | + |
| 14 | +1. **Emit** — each `codeanalyzer-*` backend projects its analysis into a **Neo4j property graph** instead of a JSON file. This is the expensive, batchable step; run it once per project (and incrementally thereafter), wherever you have compute. |
| 15 | +2. **Poll** — the CLDK SDK connects to that graph as a **read-only Cypher client**. No source is parsed at query time; the `analysis` object answers from the graph. This is the cheap, horizontally-scalable step that your agents run. |
| 16 | + |
| 17 | +Because every language's graph shares one database, agents get **multi-lingual, multi-project program analysis** behind the same `analysis` API they already use. |
| 18 | + |
| 19 | +```mermaid |
| 20 | +flowchart LR |
| 21 | + subgraph Emit["Emit · batch jobs (write once)"] |
| 22 | + J["codeanalyzer-java<br/>--emit neo4j"] |
| 23 | + P["canpy<br/>--emit neo4j"] |
| 24 | + T["cants<br/>--emit neo4j"] |
| 25 | + end |
| 26 | + subgraph DB["Shared Neo4j graph"] |
| 27 | + N[("J* · Py* · TS*<br/>one DB, many apps")] |
| 28 | + end |
| 29 | + subgraph Poll["Poll · agents (read many)"] |
| 30 | + A1["CLDK.java(backend=Neo4j…)"] |
| 31 | + A2["CLDK.python(backend=Neo4j…)"] |
| 32 | + A3["CLDK.typescript(backend=Neo4j…)"] |
| 33 | + end |
| 34 | + J --> N |
| 35 | + P --> N |
| 36 | + T --> N |
| 37 | + N --> A1 |
| 38 | + N --> A2 |
| 39 | + N --> A3 |
| 40 | +``` |
| 41 | + |
| 42 | +<Aside type="tip" title="Where this runs"> |
| 43 | +The two phases are independent processes. The emit jobs typically run on a schedule (a CI pipeline or a Kubernetes `CronJob`); the agents run as long-lived services that only read. See [Deploy on Kubernetes](/at-scale/kubernetes/) for a concrete topology, with Odoo as the worked example. |
| 44 | +</Aside> |
| 45 | + |
| 46 | +## One graph, every language, every project |
| 47 | + |
| 48 | +Each backend writes a **namespaced** label set so multiple languages share a single database without collisions. Java labels are `J*` / `J_*`, Python `Py*` / `PY_*`, TypeScript `TS*` / `TS_*`: |
| 49 | + |
| 50 | +| Language | App anchor | Module node | Symbol node (merge label) | Call edge | |
| 51 | +| --- | --- | --- | --- | --- | |
| 52 | +| Java | `:JApplication` | `:JCompilationUnit` | `:JSymbol` → `:JType` / `:JCallable` | `:J_CALLS` | |
| 53 | +| Python | `:PyApplication` | `:PyModule` | `:PySymbol` → `:PyClass` / `:PyCallable` / `:PyExternal` | `:PY_CALLS` | |
| 54 | +| TypeScript | `:TSApplication` | `:TSModule` | `:TSSymbol` → `:TSClass` … `:TSExternal` | `:TS_CALLS` | |
| 55 | + |
| 56 | +Within a language, **multiple projects** coexist too. Every node is scoped to an application anchor identified by its `--app-name`, so one database can hold `payments-service`, `web-frontend`, and `billing-core` side by side. On the read side, `application_name` selects which one a query sees. |
| 57 | + |
| 58 | +<Aside type="note" title="Languages supported"> |
| 59 | +The Neo4j emit/poll path is implemented for **Java, Python, and TypeScript**. C is analyzed in-process only — it has no Neo4j backend. |
| 60 | +</Aside> |
| 61 | + |
| 62 | +## Phase 1 — Emit a graph |
| 63 | + |
| 64 | +Every backend takes the same `--emit neo4j` flag. With `--neo4j-uri` set it pushes to a live database over Bolt; without it, it writes a self-contained `graph.cypher` snapshot you can load with `cypher-shell`. Connection settings also read the `NEO4J_URI`, `NEO4J_USERNAME`, `NEO4J_PASSWORD`, and `NEO4J_DATABASE` environment variables; an explicit flag wins over the environment. |
| 65 | + |
| 66 | +<Tabs syncKey="lang"> |
| 67 | + <TabItem label="Java"> |
| 68 | + ```bash title="Java → Neo4j (live Bolt push)" |
| 69 | + # -a 2 includes call edges (:J_CALLS); -a 1 is symbol table only. |
| 70 | + codeanalyzer -i ./payments-service -a 2 --emit neo4j \ |
| 71 | + --app-name payments-service \ |
| 72 | + --neo4j-uri bolt://localhost:7687 \ |
| 73 | + --neo4j-user neo4j --neo4j-password "$NEO4J_PASSWORD" |
| 74 | + ``` |
| 75 | + |
| 76 | + ```bash title="Java → graph.cypher snapshot (no live DB)" |
| 77 | + codeanalyzer -i ./payments-service -a 2 --emit neo4j -o ./out |
| 78 | + cypher-shell -u neo4j -p "$NEO4J_PASSWORD" < ./out/graph.cypher |
| 79 | + ``` |
| 80 | + </TabItem> |
| 81 | + <TabItem label="Python"> |
| 82 | + ```bash title="Python → Neo4j (live Bolt push)" |
| 83 | + canpy -i ./billing-core --emit neo4j \ |
| 84 | + --app-name billing-core \ |
| 85 | + --neo4j-uri bolt://localhost:7687 \ |
| 86 | + --neo4j-user neo4j --neo4j-password "$NEO4J_PASSWORD" |
| 87 | + ``` |
| 88 | + |
| 89 | + ```bash title="Python → graph.cypher snapshot" |
| 90 | + canpy -i ./billing-core --emit neo4j -o ./out # -> ./out/graph.cypher |
| 91 | + cypher-shell -u neo4j -p "$NEO4J_PASSWORD" < ./out/graph.cypher |
| 92 | + ``` |
| 93 | + </TabItem> |
| 94 | + <TabItem label="TypeScript"> |
| 95 | + ```bash title="TypeScript → Neo4j (live Bolt push)" |
| 96 | + cants -i ./web-frontend -a 2 --emit neo4j \ |
| 97 | + --app-name web-frontend \ |
| 98 | + --neo4j-uri bolt://localhost:7687 \ |
| 99 | + --neo4j-user neo4j --neo4j-password "$NEO4J_PASSWORD" |
| 100 | + ``` |
| 101 | + |
| 102 | + ```bash title="TypeScript → graph.cypher snapshot" |
| 103 | + cants -i ./web-frontend -a 2 --emit neo4j -o ./out # -> ./out/graph.cypher |
| 104 | + cypher-shell -u neo4j -p "$NEO4J_PASSWORD" < ./out/graph.cypher |
| 105 | + ``` |
| 106 | + </TabItem> |
| 107 | +</Tabs> |
| 108 | + |
| 109 | +### Writes are idempotent and incremental |
| 110 | + |
| 111 | +Re-running a job is safe and cheap, which is what makes a scheduled fleet practical: |
| 112 | + |
| 113 | +- **Idempotent** — both writers create the schema constraints and indexes first, then upsert with `MERGE` (never blind `CREATE`). Re-emitting the same project produces the same graph. |
| 114 | +- **Incremental** — a live Bolt push diffs each module against the graph by content hash and rewrites **only what changed**. On a full run, modules whose source file disappeared are pruned, scoped to that application's anchor so a shared database stays consistent. |
| 115 | +- **Index-backed** — constraints exist before any `MERGE`, so every upsert is an index seek rather than a scan, even as the database grows. |
| 116 | + |
| 117 | +### A versioned schema contract |
| 118 | + |
| 119 | +Each backend can emit its graph schema as a machine-readable, version-stamped contract with `--emit schema` (no project needed). The `schema_version` is also stamped on every graph's `:*Application` node, so a consumer can check compatibility before querying. |
| 120 | + |
| 121 | +```bash title="Export the schema contract" |
| 122 | +codeanalyzer --emit schema -o ./out # -> ./out/schema.neo4j.json |
| 123 | +canpy --emit schema -o ./out # -> ./out/schema.json |
| 124 | +cants --emit schema -o ./out # -> ./out/schema.json |
| 125 | +``` |
| 126 | + |
| 127 | +## Phase 2 — Poll the graph |
| 128 | + |
| 129 | +On the read side, nothing about the `analysis` API changes. You pass a `Neo4jConnectionConfig` as `backend=` and the SDK selects the read-only Cypher backend by config type. `project_path` is **optional** in this mode (no source is read); `application_name` selects which project in the database the queries see. |
| 130 | + |
| 131 | +<Tabs syncKey="lang"> |
| 132 | + <TabItem label="Java"> |
| 133 | + ```python title="Query a Java graph" |
| 134 | + from cldk import CLDK |
| 135 | + from cldk.analysis.commons.backend_config import Neo4jConnectionConfig |
| 136 | + |
| 137 | + analysis = CLDK.java( |
| 138 | + backend=Neo4jConnectionConfig( |
| 139 | + uri="bolt://neo4j:7687", |
| 140 | + username="reader", # read-only credentials are enough |
| 141 | + password="…", |
| 142 | + application_name="payments-service", |
| 143 | + ), |
| 144 | + ) |
| 145 | + |
| 146 | + classes = analysis.get_classes() # -> dict[str, JType], straight from the graph |
| 147 | + cg = analysis.get_call_graph() # -> networkx.DiGraph |
| 148 | + ``` |
| 149 | + </TabItem> |
| 150 | + <TabItem label="Python"> |
| 151 | + ```python title="Query a Python graph" |
| 152 | + from cldk import CLDK |
| 153 | + from cldk.analysis.commons.backend_config import Neo4jConnectionConfig |
| 154 | + |
| 155 | + analysis = CLDK.python( |
| 156 | + backend=Neo4jConnectionConfig( |
| 157 | + uri="bolt://neo4j:7687", |
| 158 | + username="reader", |
| 159 | + password="…", |
| 160 | + application_name="billing-core", |
| 161 | + ), |
| 162 | + ) |
| 163 | + |
| 164 | + callers = analysis.get_callers("billing_core.invoice.Invoice", "finalize") |
| 165 | + ``` |
| 166 | + </TabItem> |
| 167 | + <TabItem label="TypeScript"> |
| 168 | + ```python title="Query a TypeScript graph" |
| 169 | + from cldk import CLDK |
| 170 | + from cldk.analysis.commons.backend_config import Neo4jConnectionConfig |
| 171 | + |
| 172 | + analysis = CLDK.typescript( |
| 173 | + backend=Neo4jConnectionConfig( |
| 174 | + uri="bolt://neo4j:7687", |
| 175 | + username="reader", |
| 176 | + password="…", |
| 177 | + application_name="web-frontend", |
| 178 | + ), |
| 179 | + ) |
| 180 | + |
| 181 | + cg = analysis.get_call_graph() # -> networkx.DiGraph |
| 182 | + ``` |
| 183 | + </TabItem> |
| 184 | +</Tabs> |
| 185 | + |
| 186 | +The Neo4j backends (`JNeo4jBackend`, `PyNeo4jBackend`, `TSNeo4jBackend`) expose the **same method surface** as the in-process backends, so existing query code is a drop-in: only the `backend=` argument changes. |
| 187 | + |
| 188 | +<Steps> |
| 189 | +1. Install the optional driver: `pip install cldk[neo4j]` (it pulls `neo4j>=5.14`). The driver is an extra, not a core dependency. |
| 190 | +2. Point `Neo4jConnectionConfig.uri` at your Bolt endpoint and set `application_name` to the project you want to query. |
| 191 | +3. Call the usual `get_classes`, `get_call_graph`, `get_callers`, `get_callees`, and related methods. The graph answers; no source is parsed. |
| 192 | +</Steps> |
| 193 | + |
| 194 | +<Aside type="caution" title="What “at scale” does and doesn’t mean here"> |
| 195 | +Scale here comes from **architecture**, not tuning knobs: idempotent index-backed `MERGE`, content-hash incrementality, and the freedom to fan emit jobs out horizontally. The writers do **not** expose a configurable batch size or `PERIODIC COMMIT`, and a **targeted run** (`--target-files`) skips orphan pruning, so deleted files are only cleaned up on a full run. Plan your jobs as periodic full runs with incremental pushes in between. |
| 196 | +</Aside> |
| 197 | + |
| 198 | +## See also |
| 199 | + |
| 200 | +<CardGrid> |
| 201 | + <LinkCard title="Deploy on Kubernetes" href="/at-scale/kubernetes/" description="A Job/CronJob emit topology and the Odoo multi-lingual walkthrough." /> |
| 202 | + <LinkCard title="codeanalyzer-java" href="/backends/codeanalyzer-java/" description="The backend that emits the J* graph." /> |
| 203 | + <LinkCard title="codeanalyzer-python" href="/backends/codeanalyzer-python/" description="The backend that emits the Py* graph." /> |
| 204 | + <LinkCard title="codeanalyzer-ts" href="/backends/codeanalyzer-ts/" description="The backend that emits the TS* graph." /> |
| 205 | +</CardGrid> |
0 commit comments