Skip to content

Commit a9668ec

Browse files
committed
docs: add "analysis at scale" (Neo4j) section and sync TypeScript status
Add a top-level "At scale" section documenting the emit/poll architecture: codeanalyzer-{java,python,typescript} project a namespaced property graph into a shared Neo4j (--emit neo4j), and the CLDK SDK polls it as a read-only Cypher client via Neo4jConnectionConfig. Covers the shared-graph schema, idempotent incremental writes, the --emit schema contract, a Kubernetes CronJob/Deployment topology, and an end-to-end Odoo (Python + JS) multi-lingual walkthrough. Reconcile the contradictory TypeScript maturity claims to a single voice: TS is beta — symbol table and call graph work through CLDK.typescript(...), entrypoint detection is not yet implemented. Update what-is-cldk, the Python-API reference, the backends overview, and codeanalyzer-ts. Cross-link existing read-only Neo4j notes to the new section and correct the Neo4j language list to Java/Python/TS.
1 parent 902d98a commit a9668ec

10 files changed

Lines changed: 427 additions & 11 deletions

File tree

astro.config.mjs

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,13 @@ export default defineConfig({
169169
{ label: "Java examples", slug: "examples/java", attrs: { "data-cldk-icon": "java" } },
170170
],
171171
},
172+
{
173+
label: "At scale",
174+
items: [
175+
{ label: "Analysis at scale", slug: "at-scale", attrs: { "data-cldk-icon": "globe-network-16" } },
176+
{ label: "Deploy on Kubernetes", slug: "at-scale/kubernetes", attrs: { "data-cldk-icon": "cloud-16" } },
177+
],
178+
},
172179
{
173180
label: "SDKs",
174181
items: [
Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
---
2+
title: Multi-lingual, multi-project analysis at scale
3+
description: Analyze many projects in many languages once, push the results into a shared Neo4j graph, and let agents query all of it through one read-only analysis API.
4+
---
5+
6+
import { Tabs, TabItem, Steps, Aside, LinkCard, CardGrid, Badge } from "@astrojs/starlight/components";
7+
8+
In the [default workflow](/quickstart/), CLDK runs the analysis in-process: you point `CLDK.java(...)` at a project, the backend parses it, and the typed models live in memory for the lifetime of that object. That is the right model for a single project on a single machine.
9+
10+
It does not fit a fleet. When you have hundreds of repositories across several languages, and agents that need to answer structural questions about any of them at any time, re-analyzing on every request is wasteful, and holding every project in memory is impossible.
11+
12+
CLDK supports a second model for exactly this case. **Analysis and querying are split into two phases that scale independently:**
13+
14+
1. **Emit** — each `codeanalyzer-*` backend projects its analysis into a **Neo4j property graph** instead of a JSON file. This is the expensive, batchable step; run it once per project (and incrementally thereafter), wherever you have compute.
15+
2. **Poll** — the CLDK SDK connects to that graph as a **read-only Cypher client**. No source is parsed at query time; the `analysis` object answers from the graph. This is the cheap, horizontally-scalable step that your agents run.
16+
17+
Because every language's graph shares one database, agents get **multi-lingual, multi-project program analysis** behind the same `analysis` API they already use.
18+
19+
```mermaid
20+
flowchart LR
21+
subgraph Emit["Emit · batch jobs (write once)"]
22+
J["codeanalyzer-java<br/>--emit neo4j"]
23+
P["canpy<br/>--emit neo4j"]
24+
T["cants<br/>--emit neo4j"]
25+
end
26+
subgraph DB["Shared Neo4j graph"]
27+
N[("J* · Py* · TS*<br/>one DB, many apps")]
28+
end
29+
subgraph Poll["Poll · agents (read many)"]
30+
A1["CLDK.java(backend=Neo4j…)"]
31+
A2["CLDK.python(backend=Neo4j…)"]
32+
A3["CLDK.typescript(backend=Neo4j…)"]
33+
end
34+
J --> N
35+
P --> N
36+
T --> N
37+
N --> A1
38+
N --> A2
39+
N --> A3
40+
```
41+
42+
<Aside type="tip" title="Where this runs">
43+
The two phases are independent processes. The emit jobs typically run on a schedule (a CI pipeline or a Kubernetes `CronJob`); the agents run as long-lived services that only read. See [Deploy on Kubernetes](/at-scale/kubernetes/) for a concrete topology, with Odoo as the worked example.
44+
</Aside>
45+
46+
## One graph, every language, every project
47+
48+
Each backend writes a **namespaced** label set so multiple languages share a single database without collisions. Java labels are `J*` / `J_*`, Python `Py*` / `PY_*`, TypeScript `TS*` / `TS_*`:
49+
50+
| Language | App anchor | Module node | Symbol node (merge label) | Call edge |
51+
| --- | --- | --- | --- | --- |
52+
| Java | `:JApplication` | `:JCompilationUnit` | `:JSymbol``:JType` / `:JCallable` | `:J_CALLS` |
53+
| Python | `:PyApplication` | `:PyModule` | `:PySymbol``:PyClass` / `:PyCallable` / `:PyExternal` | `:PY_CALLS` |
54+
| TypeScript | `:TSApplication` | `:TSModule` | `:TSSymbol``:TSClass``:TSExternal` | `:TS_CALLS` |
55+
56+
Within a language, **multiple projects** coexist too. Every node is scoped to an application anchor identified by its `--app-name`, so one database can hold `payments-service`, `web-frontend`, and `billing-core` side by side. On the read side, `application_name` selects which one a query sees.
57+
58+
<Aside type="note" title="Languages supported">
59+
The Neo4j emit/poll path is implemented for **Java, Python, and TypeScript**. C is analyzed in-process only — it has no Neo4j backend.
60+
</Aside>
61+
62+
## Phase 1 — Emit a graph
63+
64+
Every backend takes the same `--emit neo4j` flag. With `--neo4j-uri` set it pushes to a live database over Bolt; without it, it writes a self-contained `graph.cypher` snapshot you can load with `cypher-shell`. Connection settings also read the `NEO4J_URI`, `NEO4J_USERNAME`, `NEO4J_PASSWORD`, and `NEO4J_DATABASE` environment variables; an explicit flag wins over the environment.
65+
66+
<Tabs syncKey="lang">
67+
<TabItem label="Java">
68+
```bash title="Java → Neo4j (live Bolt push)"
69+
# -a 2 includes call edges (:J_CALLS); -a 1 is symbol table only.
70+
codeanalyzer -i ./payments-service -a 2 --emit neo4j \
71+
--app-name payments-service \
72+
--neo4j-uri bolt://localhost:7687 \
73+
--neo4j-user neo4j --neo4j-password "$NEO4J_PASSWORD"
74+
```
75+
76+
```bash title="Java → graph.cypher snapshot (no live DB)"
77+
codeanalyzer -i ./payments-service -a 2 --emit neo4j -o ./out
78+
cypher-shell -u neo4j -p "$NEO4J_PASSWORD" < ./out/graph.cypher
79+
```
80+
</TabItem>
81+
<TabItem label="Python">
82+
```bash title="Python → Neo4j (live Bolt push)"
83+
canpy -i ./billing-core --emit neo4j \
84+
--app-name billing-core \
85+
--neo4j-uri bolt://localhost:7687 \
86+
--neo4j-user neo4j --neo4j-password "$NEO4J_PASSWORD"
87+
```
88+
89+
```bash title="Python → graph.cypher snapshot"
90+
canpy -i ./billing-core --emit neo4j -o ./out # -> ./out/graph.cypher
91+
cypher-shell -u neo4j -p "$NEO4J_PASSWORD" < ./out/graph.cypher
92+
```
93+
</TabItem>
94+
<TabItem label="TypeScript">
95+
```bash title="TypeScript → Neo4j (live Bolt push)"
96+
cants -i ./web-frontend -a 2 --emit neo4j \
97+
--app-name web-frontend \
98+
--neo4j-uri bolt://localhost:7687 \
99+
--neo4j-user neo4j --neo4j-password "$NEO4J_PASSWORD"
100+
```
101+
102+
```bash title="TypeScript → graph.cypher snapshot"
103+
cants -i ./web-frontend -a 2 --emit neo4j -o ./out # -> ./out/graph.cypher
104+
cypher-shell -u neo4j -p "$NEO4J_PASSWORD" < ./out/graph.cypher
105+
```
106+
</TabItem>
107+
</Tabs>
108+
109+
### Writes are idempotent and incremental
110+
111+
Re-running a job is safe and cheap, which is what makes a scheduled fleet practical:
112+
113+
- **Idempotent** — both writers create the schema constraints and indexes first, then upsert with `MERGE` (never blind `CREATE`). Re-emitting the same project produces the same graph.
114+
- **Incremental** — a live Bolt push diffs each module against the graph by content hash and rewrites **only what changed**. On a full run, modules whose source file disappeared are pruned, scoped to that application's anchor so a shared database stays consistent.
115+
- **Index-backed** — constraints exist before any `MERGE`, so every upsert is an index seek rather than a scan, even as the database grows.
116+
117+
### A versioned schema contract
118+
119+
Each backend can emit its graph schema as a machine-readable, version-stamped contract with `--emit schema` (no project needed). The `schema_version` is also stamped on every graph's `:*Application` node, so a consumer can check compatibility before querying.
120+
121+
```bash title="Export the schema contract"
122+
codeanalyzer --emit schema -o ./out # -> ./out/schema.neo4j.json
123+
canpy --emit schema -o ./out # -> ./out/schema.json
124+
cants --emit schema -o ./out # -> ./out/schema.json
125+
```
126+
127+
## Phase 2 — Poll the graph
128+
129+
On the read side, nothing about the `analysis` API changes. You pass a `Neo4jConnectionConfig` as `backend=` and the SDK selects the read-only Cypher backend by config type. `project_path` is **optional** in this mode (no source is read); `application_name` selects which project in the database the queries see.
130+
131+
<Tabs syncKey="lang">
132+
<TabItem label="Java">
133+
```python title="Query a Java graph"
134+
from cldk import CLDK
135+
from cldk.analysis.commons.backend_config import Neo4jConnectionConfig
136+
137+
analysis = CLDK.java(
138+
backend=Neo4jConnectionConfig(
139+
uri="bolt://neo4j:7687",
140+
username="reader", # read-only credentials are enough
141+
password="",
142+
application_name="payments-service",
143+
),
144+
)
145+
146+
classes = analysis.get_classes() # -> dict[str, JType], straight from the graph
147+
cg = analysis.get_call_graph() # -> networkx.DiGraph
148+
```
149+
</TabItem>
150+
<TabItem label="Python">
151+
```python title="Query a Python graph"
152+
from cldk import CLDK
153+
from cldk.analysis.commons.backend_config import Neo4jConnectionConfig
154+
155+
analysis = CLDK.python(
156+
backend=Neo4jConnectionConfig(
157+
uri="bolt://neo4j:7687",
158+
username="reader",
159+
password="",
160+
application_name="billing-core",
161+
),
162+
)
163+
164+
callers = analysis.get_callers("billing_core.invoice.Invoice", "finalize")
165+
```
166+
</TabItem>
167+
<TabItem label="TypeScript">
168+
```python title="Query a TypeScript graph"
169+
from cldk import CLDK
170+
from cldk.analysis.commons.backend_config import Neo4jConnectionConfig
171+
172+
analysis = CLDK.typescript(
173+
backend=Neo4jConnectionConfig(
174+
uri="bolt://neo4j:7687",
175+
username="reader",
176+
password="",
177+
application_name="web-frontend",
178+
),
179+
)
180+
181+
cg = analysis.get_call_graph() # -> networkx.DiGraph
182+
```
183+
</TabItem>
184+
</Tabs>
185+
186+
The Neo4j backends (`JNeo4jBackend`, `PyNeo4jBackend`, `TSNeo4jBackend`) expose the **same method surface** as the in-process backends, so existing query code is a drop-in: only the `backend=` argument changes.
187+
188+
<Steps>
189+
1. Install the optional driver: `pip install cldk[neo4j]` (it pulls `neo4j>=5.14`). The driver is an extra, not a core dependency.
190+
2. Point `Neo4jConnectionConfig.uri` at your Bolt endpoint and set `application_name` to the project you want to query.
191+
3. Call the usual `get_classes`, `get_call_graph`, `get_callers`, `get_callees`, and related methods. The graph answers; no source is parsed.
192+
</Steps>
193+
194+
<Aside type="caution" title="What “at scale” does and doesn’t mean here">
195+
Scale here comes from **architecture**, not tuning knobs: idempotent index-backed `MERGE`, content-hash incrementality, and the freedom to fan emit jobs out horizontally. The writers do **not** expose a configurable batch size or `PERIODIC COMMIT`, and a **targeted run** (`--target-files`) skips orphan pruning, so deleted files are only cleaned up on a full run. Plan your jobs as periodic full runs with incremental pushes in between.
196+
</Aside>
197+
198+
## See also
199+
200+
<CardGrid>
201+
<LinkCard title="Deploy on Kubernetes" href="/at-scale/kubernetes/" description="A Job/CronJob emit topology and the Odoo multi-lingual walkthrough." />
202+
<LinkCard title="codeanalyzer-java" href="/backends/codeanalyzer-java/" description="The backend that emits the J* graph." />
203+
<LinkCard title="codeanalyzer-python" href="/backends/codeanalyzer-python/" description="The backend that emits the Py* graph." />
204+
<LinkCard title="codeanalyzer-ts" href="/backends/codeanalyzer-ts/" description="The backend that emits the TS* graph." />
205+
</CardGrid>

0 commit comments

Comments
 (0)