Graph extension (Proj. Manifold Stage 1/2)#4996
Closed
Hextaku wants to merge 1 commit into
Closed
Conversation
- Add openCypher parser crate for parsing Cypher queries - Implement Cypher-to-RelExpr translator with fixed-depth and variable-length path support - Add graph query execution via PostgreSQL wire protocol - Implement multiple MATCH pattern / cross-join support and NOT expression support - Add graph query subscription integration with compile_cypher entry point - Add userland graph module with Vertex/Edge tables and CRUD reducers - Implement graph traversal reducers (BFS, DFS, shortest-path) with benchmarks - Add graph-algo crate with traversal benchmark harness - Add graph module integration and smoke tests
|
|
Contributor
|
@Hextaku Thank you for your contribution! We don't have the bandwidth to maintain this at present, but we can definitely use it as future reference! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a userland graph extension to SpacetimeDB: a new
GraphIdSATS type, a sharedgraph-algocrate with BFS/DFS/shortest-path traversal algorithms, and agraph-moduleWasm cdylib with vertex/edge CRUD, traversal reducers, and procedure-based optimized queries. Zero engine changes — everything runs through the existing reducer/procedure API.Full context: Project Manifold Wiki
What changes
crates/sats/src/graph_id.rs(new),algebraic_type.rs,product_type.rs,satn.rs,lib.rscrates/lib/src/lib.rs,crates/expr/src/lib.rs,crates/pg/src/encoder.rsmodules/graph-algo/(new crate, 310 lines + 193 line bench)modules/graph/(new crate, 440 lines)Cargo.toml(+2 members),Cargo.lockWhy
I'm working on a project that needs native graph capabilities, and SpacetimeDB's real-time subscription architecture would be a perfect fit — it would dramatically reduce scope compared to running a separate graph database and managing server-db sync at scale. The alternative (Apache AGE) introduces significant operational complexity without a clear path to solving the sync problem cleanly.
I reached out on Discord asking whether graph features would be accepted in PRs. While I didn't get a direct response from Clockwork Labs, nobody shut the idea down either. Rather than wait indefinitely, I decided to build it as a proof of concept — a userland module that adds real graph capabilities with zero engine changes, backed by cross-system benchmarks that characterize exactly how far you can get without touching the engine.
The goal is to demonstrate that:
Graph support is viable in SpacetimeDB today, with no architectural changes.
The implementation is minimal and follows existing patterns (the SATS type mirrors Identity/Timestamp/Uuid exactly).
The userland ceiling is well understood — and where it ends (large-graph shortest path), Stage 2's case for engine-level operators is backed by measured data, not speculation.
If this is something Clockwork Labs is interested in merging: great!
If not: the benchmarks are public, the module works, the data speaks for itself, and anybody is able to deploy this for their projects.
I'd really enjoyed the challenge and am at least going to take a look if I can also pull off Stage 2.
More detail: Stage 1 — Userland Graph Extension
Benchmark highlights
Benchmarks are NOT on this branch or part of the MR. But on https://github.com/Hextaku/SpacetimeDB-Manifold/tree/pr/manifold-stage-1-bench
SpacetimeDB's userland module (Wasm, zero engine changes) vs. Neo4j and Apache AGE on three SNAP datasets:
ego-Facebook (~4K vertices, ~88K edges):
com-DBLP (~317K vertices, ~1M edges):
com-LiveJournal (~4M vertices, ~35M edges):
SpacetimeDB wins loading (4-6x), neighbors (constant ~1.5 ms via index scan), and BFS full traversal on every dataset. Neo4j leads shortest path (30-62x) due to its native bidirectional algorithm — a gap Stage 2 is designed to close.
Full results: Benchmarks
What's next: Stage 2
Stage 2 moves from userland to the SpacetimeDB engine. The benchmark data makes the priorities clear:
More detail: Stage 2 — Native Graph Engine
Test state
Known limitations
AlgebraicTypeDef/TypespaceForGenerate) not yet implemented — blocked on client-side graph type design (Stage 2).