You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: add Array engine and promote eight-engine architecture
Add Array engine documentation and bitemporal query reference.
Promote Timeseries and Spatial from "columnar profiles" to peer engines
with their own storage-engines/ pages. Update all cross-references,
engine counts, and the sidebar to reflect eight peer engines plus Graph
and FTS as cross-engine overlays.
Expand vector-search reference with quantization codec comparison table,
index implementation details (Vamana, NaviX, SIEVE, MetaEmbed, SPFresh,
Matryoshka), and vector-primary collection DDL. Add cross-engine identity
section to the architecture overview.
Copy file name to clipboardExpand all lines: docs/architecture/overview.rdx
+17Lines changed: 17 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -55,3 +55,20 @@ nodedb-client / FFI / WASM
55
55
```
56
56
57
57
Both paths produce the same plan and execute identically on the Data Plane.
58
+
59
+
## Cross-Engine Identity
60
+
61
+
Every row in every engine — document, KV, columnar, timeseries, spatial, vector, array, graph node, FTS posting — carries a stable global `u32` **surrogate** allocated at insert from a WAL-durable, Raft-replicated monotonic counter. Every engine keys its internal indexes on the surrogate, so cross-engine prefilter and join reduce to roaring-bitmap intersections with **zero per-query translation**.
62
+
63
+
A query like "find product cells whose embedding is near `$q`, that have FTS hits for `'memory leak'`, that live within 5km of point P, in tenant 42" turns into:
64
+
65
+
```
66
+
vector_index.search($q, k) → roaring bitmap A
67
+
fts_index.match("memory leak") → roaring bitmap B
68
+
spatial_index.dwithin(P, 5km) → roaring bitmap C
69
+
metadata_index.tenant_id = 42 → roaring bitmap D
70
+
71
+
A ∩ B ∩ C ∩ D → final candidate surrogate set
72
+
```
73
+
74
+
No `HashMap<surrogate, doc_id>` translations between hops. Adding a new engine does not require new translation paths — it just allocates surrogates from the same counter. This is the mechanism behind every cross-engine query example you'll see in the SQL reference.
Copy file name to clipboardExpand all lines: docs/crdt-sync/overview.rdx
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -33,4 +33,4 @@ Multiple devices converge to the same state regardless of operation order. Loro'
33
33
34
34
## Local Engine Capabilities
35
35
36
-
All seven engines work locally on NodeDB-Lite: vector search, graph traversal, document storage, columnar queries, KV lookups, FTS, and CRDT state — with sub-millisecond reads and no network dependency.
36
+
All eight engines work locally on NodeDB-Lite: document, KV, columnar, timeseries, spatial, vector, array, plus graph traversal and FTS overlays — with sub-millisecond reads and no network dependency.
Copy file name to clipboardExpand all lines: docs/data-modeling/collections.rdx
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -35,7 +35,7 @@ CREATE COLLECTION sessions (key TEXT PRIMARY KEY) WITH (engine='kv');
35
35
36
36
## Storage Engines
37
37
38
-
NodeDB has seven peer engines. Pick one per collection via `WITH (engine='<name>')`. The default (no `engine=`) is `document_schemaless`.
38
+
NodeDB has eight peer engines. Seven of them are picked per collection via `WITH (engine='<name>')` — the default (no `engine=`) is `document_schemaless`. The eighth, Array, uses its own `CREATE ARRAY` DDL family. See [Array Engine](../storage-engines/array) for that path.
Copy file name to clipboardExpand all lines: docs/introduction/what-is-nodedb.rdx
+7-5Lines changed: 7 additions & 5 deletions
Original file line number
Diff line number
Diff line change
@@ -1,32 +1,34 @@
1
1
---
2
2
title: What is NodeDB
3
-
description: A distributed hybrid database with seven engines for multi-modal AI and agentic workloads.
3
+
description: A distributed hybrid database with eight peer engines for multi-modal AI and agentic workloads.
4
4
---
5
5
6
6
# What is NodeDB
7
7
8
-
NodeDB is a single Rust binary that provides seven data engines — Vector, Graph, Document (schemaless + strict), Columnar (with Timeseries and Spatial profiles), Key-Value, and Full-Text Search — each built with purpose-specific data structures. All engines share the same storage, memory, and query planner. Cross-engine queries execute in one process with zero network hops.
8
+
NodeDB is a single Rust binary that provides eight peer engines — Document (schemaless), Document (strict), Key-Value, Columnar, Timeseries, Spatial, Vector, and Array — sharing one storage core, plus Graph and Full-Text Search as cross-engine overlays on any collection. Seven of the eight are selected per-collection via `WITH (engine='<name>')`; Array uses its own `CREATE ARRAY` DDL family. Each engine is built with purpose-specific data structures. All share the same storage, memory, and query planner. Cross-engine queries execute in one process with zero network hops.
9
9
10
10
## The Problem
11
11
12
12
Modern applications don't fit in one database. A healthcare app needs patient records (relational), medical imaging embeddings (vector), care team relationships (graph), device telemetry (timeseries), and offline-first sync for field workers (CRDT). The industry answer is a polyglot stack: PostgreSQL for relational, Qdrant for vectors, Neo4j for graphs, ClickHouse for timeseries, Redis for caching. Each system has its own protocol, deployment, and failure modes. Cross-database queries require application-level joins.
13
13
14
14
Some databases claim to solve this by bolting on capabilities — "graph" as recursive JOINs, "timeseries" without columnar compression or continuous aggregation. The features exist in name but not in performance.
15
15
16
-
## Seven Engines
16
+
## Eight Engines
17
17
18
18
**Vector** — HNSW index with SQ8, PQ, and IVF-PQ quantization. SIMD-accelerated distance math. Adaptive bitmap pre-filtering.
19
19
20
20
**Graph** — CSR adjacency index with 13 native algorithms (PageRank, WCC, Louvain, SSSP, etc.) and a Cypher-subset MATCH pattern engine. GraphRAG fusion with vector search.
21
21
22
22
**Document** — Two modes per collection. Schemaless: MessagePack blobs with CRDT sync. Strict: Binary Tuples with O(1) field extraction and 3-4x cache density over BSON.
**Columnar / Timeseries / Spatial** — Three peer engines sharing one compressed-column storage core (ALP, FastLanes, FSST, Gorilla, LZ4 codecs; block statistics; predicate pushdown). `columnar` for general analytics; `timeseries` adds append-only ingest, retention, and continuous aggregation; `spatial` adds R*-tree, geohash, and H3 indexing.
25
25
26
26
**Key-Value** — Hash-indexed O(1) point lookups with typed value fields. Native TTL, secondary indexes, atomic INCR/CAS, and predicate-filtered scans. SQL-queryable and joinable.
**Array** — ND sparse multi-dimensional engine with tile-based storage, Z-order indexing, per-tile MBR statistics, and bitemporal cells. Replaces TileDB / Zarr / SciDB / Rasdaman for genomics, single-cell biology, raster cubes, and climate models.
31
+
30
32
**CRDT** — Loro-backed conflict-free replicated data types. AP on the edge, CP in the cloud. SQL constraint validation at sync time with compensation hints.
31
33
32
34
## Three Deployment Modes
@@ -35,7 +37,7 @@ Some databases claim to solve this by bolting on capabilities — "graph" as rec
35
37
36
38
**Origin (local)** — Same binary, single-node. No cluster overhead. Like running PostgreSQL locally.
37
39
38
-
**NodeDB-Lite (embedded)** — In-process library for phones, browsers (WASM), and desktops. All seven engines run locally with sub-millisecond reads. CRDT sync to Origin via WebSocket.
40
+
**NodeDB-Lite (embedded)** — In-process library for phones, browsers (WASM), and desktops. All eight engines run locally with sub-millisecond reads. CRDT sync to Origin via WebSocket.
The Timeseries profile uses `timeseries_budget_fraction` (default 10%).
33
+
The `timeseries` peer engine has its own memory budget knob `timeseries_budget_fraction` (default 10%); `columnar` and `spatial` draw from the columnar engine pool.
The cost-model planner (`target_recall`) will pick `oversample` and `ef_search` automatically once you set the recall target. Manually set those two only when you need a hard latency ceiling.
88
+
89
+
## Index Implementations
90
+
91
+
You don't pick the underlying index directly — it's chosen by the planner from collection metadata + workload signals:
92
+
93
+
- **HNSW** — in-memory hierarchical graph, the default for moderate-size indexes
94
+
- **Vamana / DiskANN** — flat-beam SSD-resident graph for billion-scale on a single node (`Tier 2` of the vector frontier)
95
+
- **NaviX adaptive-local filtered traversal** (VLDB 2025) — switches per-hop between standard / directed / blind heuristics based on local selectivity. Replaces classic ACORN-1 filtered ANN.
96
+
- **SIEVE workload-driven subindex collections** — the planner builds specialized HNSW subindices for stable predicates (e.g. `tenant_id`) and routes filtered queries to them.
97
+
- **MetaEmbed multi-vector + ColBERT MaxSim + PLAID** (ICLR 2026) — learnable Meta Tokens replace per-token explosion; budgeted MaxSim at query time via `meta_token_budget`.
98
+
- **Matryoshka adaptive-dim querying** — coarse-to-fine ranking on the first-N dimensions of MRL embeddings via `query_dim`.
99
+
- **SPFresh streaming updates** (SOSP 2023) — LIRE topology-aware local rebalancing; no full-rebuild stalls when vectors are added/removed.
100
+
101
+
## Vector-Primary Collections
102
+
103
+
By default, vectors are an *index* attached to a column on a normal collection — the document/strict store is the source of truth, and the vector index is a side path. For pure-vector workloads (RAG corpora, recommendation memory, embedding stores) you can flip a collection into **vector-primary** mode, where the vector index becomes the primary access path and the document store is a metadata sidecar:
| `payload_indexes` | Per-field equality / range / boolean indexes over the metadata sidecar for filtered ANN. Replaces Pinecone metadata filters. |
133
+
134
+
In vector-primary mode the planner treats the vector index as the source of truth for IDs; metadata fetches only happen for hit IDs. Cross-engine queries, CRDT sync, and SQL semantics all keep working — `primary='vector'` is purely an *access-path* hint, not a different engine.
135
+
136
+
Default `primary='document'` is unchanged: the existing `CREATE VECTOR INDEX ON ...` syntax continues to work for vector-as-side-index workloads.
0 commit comments