Skip to content

Commit 68c54a1

Browse files
committed
docs: add Array engine and promote eight-engine architecture
Add Array engine documentation and bitemporal query reference. Promote Timeseries and Spatial from "columnar profiles" to peer engines with their own storage-engines/ pages. Update all cross-references, engine counts, and the sidebar to reflect eight peer engines plus Graph and FTS as cross-engine overlays. Expand vector-search reference with quantization codec comparison table, index implementation details (Vamana, NaviX, SIEVE, MetaEmbed, SPFresh, Matryoshka), and vector-primary collection DDL. Add cross-engine identity section to the architecture overview.
1 parent 94407af commit 68c54a1

14 files changed

Lines changed: 556 additions & 44 deletions

File tree

docs/administration/rls.rdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Row-Level Security
3-
description: Per-row filtering based on authenticated user context. Works across all seven engines.
3+
description: Per-row filtering based on authenticated user context. Works across all eight engines.
44
---
55

66
# Row-Level Security (RLS)

docs/architecture/overview.rdx

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,20 @@ nodedb-client / FFI / WASM
5555
```
5656

5757
Both paths produce the same plan and execute identically on the Data Plane.
58+
59+
## Cross-Engine Identity
60+
61+
Every row in every engine — document, KV, columnar, timeseries, spatial, vector, array, graph node, FTS posting — carries a stable global `u32` **surrogate** allocated at insert from a WAL-durable, Raft-replicated monotonic counter. Every engine keys its internal indexes on the surrogate, so cross-engine prefilter and join reduce to roaring-bitmap intersections with **zero per-query translation**.
62+
63+
A query like "find product cells whose embedding is near `$q`, that have FTS hits for `'memory leak'`, that live within 5km of point P, in tenant 42" turns into:
64+
65+
```
66+
vector_index.search($q, k) → roaring bitmap A
67+
fts_index.match("memory leak") → roaring bitmap B
68+
spatial_index.dwithin(P, 5km) → roaring bitmap C
69+
metadata_index.tenant_id = 42 → roaring bitmap D
70+
71+
A ∩ B ∩ C ∩ D → final candidate surrogate set
72+
```
73+
74+
No `HashMap<surrogate, doc_id>` translations between hops. Adding a new engine does not require new translation paths — it just allocates surrogates from the same counter. This is the mechanism behind every cross-engine query example you'll see in the SQL reference.

docs/crdt-sync/offline-sync.rdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,4 +65,4 @@ await db.execute("INSERT INTO invoices ...");
6565
await db.sync("wss://origin.example.com/sync");
6666
```
6767

68-
All seven engines work locally with sub-millisecond reads. CRDT sync is transparent.
68+
All eight engines work locally with sub-millisecond reads. CRDT sync is transparent.

docs/crdt-sync/overview.rdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,4 +33,4 @@ Multiple devices converge to the same state regardless of operation order. Loro'
3333

3434
## Local Engine Capabilities
3535

36-
All seven engines work locally on NodeDB-Lite: vector search, graph traversal, document storage, columnar queries, KV lookups, FTS, and CRDT state — with sub-millisecond reads and no network dependency.
36+
All eight engines work locally on NodeDB-Lite: document, KV, columnar, timeseries, spatial, vector, array, plus graph traversal and FTS overlays — with sub-millisecond reads and no network dependency.

docs/data-modeling/collections.rdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ CREATE COLLECTION sessions (key TEXT PRIMARY KEY) WITH (engine='kv');
3535

3636
## Storage Engines
3737

38-
NodeDB has seven peer engines. Pick one per collection via `WITH (engine='<name>')`. The default (no `engine=`) is `document_schemaless`.
38+
NodeDB has eight peer engines. Seven of them are picked per collection via `WITH (engine='<name>')` — the default (no `engine=`) is `document_schemaless`. The eighth, Array, uses its own `CREATE ARRAY` DDL family. See [Array Engine](../storage-engines/array) for that path.
3939

4040
| Engine | Selector | Best for |
4141
| --------------------- | ----------------------------------- | ------------------------------------------------- |

docs/introduction/what-is-nodedb.rdx

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,34 @@
11
---
22
title: What is NodeDB
3-
description: A distributed hybrid database with seven engines for multi-modal AI and agentic workloads.
3+
description: A distributed hybrid database with eight peer engines for multi-modal AI and agentic workloads.
44
---
55

66
# What is NodeDB
77

8-
NodeDB is a single Rust binary that provides seven data engines — Vector, Graph, Document (schemaless + strict), Columnar (with Timeseries and Spatial profiles), Key-Value, and Full-Text Search — each built with purpose-specific data structures. All engines share the same storage, memory, and query planner. Cross-engine queries execute in one process with zero network hops.
8+
NodeDB is a single Rust binary that provides eight peer engines — Document (schemaless), Document (strict), Key-Value, Columnar, Timeseries, Spatial, Vector, and Array — sharing one storage core, plus Graph and Full-Text Search as cross-engine overlays on any collection. Seven of the eight are selected per-collection via `WITH (engine='<name>')`; Array uses its own `CREATE ARRAY` DDL family. Each engine is built with purpose-specific data structures. All share the same storage, memory, and query planner. Cross-engine queries execute in one process with zero network hops.
99

1010
## The Problem
1111

1212
Modern applications don't fit in one database. A healthcare app needs patient records (relational), medical imaging embeddings (vector), care team relationships (graph), device telemetry (timeseries), and offline-first sync for field workers (CRDT). The industry answer is a polyglot stack: PostgreSQL for relational, Qdrant for vectors, Neo4j for graphs, ClickHouse for timeseries, Redis for caching. Each system has its own protocol, deployment, and failure modes. Cross-database queries require application-level joins.
1313

1414
Some databases claim to solve this by bolting on capabilities — "graph" as recursive JOINs, "timeseries" without columnar compression or continuous aggregation. The features exist in name but not in performance.
1515

16-
## Seven Engines
16+
## Eight Engines
1717

1818
**Vector** — HNSW index with SQ8, PQ, and IVF-PQ quantization. SIMD-accelerated distance math. Adaptive bitmap pre-filtering.
1919

2020
**Graph** — CSR adjacency index with 13 native algorithms (PageRank, WCC, Louvain, SSSP, etc.) and a Cypher-subset MATCH pattern engine. GraphRAG fusion with vector search.
2121

2222
**Document** — Two modes per collection. Schemaless: MessagePack blobs with CRDT sync. Strict: Binary Tuples with O(1) field extraction and 3-4x cache density over BSON.
2323

24-
**Columnar** — Per-column compression (ALP, FastLanes, FSST, Gorilla, LZ4). Block statistics and predicate pushdown. Three profiles: Plain (general analytics), Timeseries (append-only, retention, continuous aggregation), Spatial (R*-tree, geohash, H3).
24+
**Columnar / Timeseries / Spatial** — Three peer engines sharing one compressed-column storage core (ALP, FastLanes, FSST, Gorilla, LZ4 codecs; block statistics; predicate pushdown). `columnar` for general analytics; `timeseries` adds append-only ingest, retention, and continuous aggregation; `spatial` adds R*-tree, geohash, and H3 indexing.
2525

2626
**Key-Value** — Hash-indexed O(1) point lookups with typed value fields. Native TTL, secondary indexes, atomic INCR/CAS, and predicate-filtered scans. SQL-queryable and joinable.
2727

2828
**Full-Text Search** — Block-Max WAND optimized BM25 with 16 Snowball stemmers, 27-language stop words, CJK bigram tokenization, posting compression, fuzzy matching, and native hybrid vector fusion.
2929

30+
**Array** — ND sparse multi-dimensional engine with tile-based storage, Z-order indexing, per-tile MBR statistics, and bitemporal cells. Replaces TileDB / Zarr / SciDB / Rasdaman for genomics, single-cell biology, raster cubes, and climate models.
31+
3032
**CRDT** — Loro-backed conflict-free replicated data types. AP on the edge, CP in the cloud. SQL constraint validation at sync time with compensation hints.
3133

3234
## Three Deployment Modes
@@ -35,7 +37,7 @@ Some databases claim to solve this by bolting on capabilities — "graph" as rec
3537

3638
**Origin (local)** — Same binary, single-node. No cluster overhead. Like running PostgreSQL locally.
3739

38-
**NodeDB-Lite (embedded)** — In-process library for phones, browsers (WASM), and desktops. All seven engines run locally with sub-millisecond reads. CRDT sync to Origin via WebSocket.
40+
**NodeDB-Lite (embedded)** — In-process library for phones, browsers (WASM), and desktops. All eight engines run locally with sub-millisecond reads. CRDT sync to Origin via WebSocket.
3941

4042
## PostgreSQL Compatible
4143

docs/reference/engine-config.rdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,9 @@ Graph, KV, and FTS engines do not have dedicated budget config fields. They draw
2828
| Memtable threshold | 32M entries | Flush to segment threshold |
2929
| Compaction levels | 8x8 | Level-based compaction |
3030

31-
## Columnar Engine
31+
## Columnar / Timeseries / Spatial Engines
3232

33-
The Timeseries profile uses `timeseries_budget_fraction` (default 10%).
33+
The `timeseries` peer engine has its own memory budget knob `timeseries_budget_fraction` (default 10%); `columnar` and `spatial` draw from the columnar engine pool.
3434

3535
| Parameter | Default | Description |
3636
| ------------ | ------- | -------------------------- |
@@ -51,5 +51,5 @@ These are the actual config fields controlling memory allocation:
5151
| `vector_budget_fraction` | 0.30 | HNSW index and vector data |
5252
| `sparse_budget_fraction` | 0.15 | Graph, KV, FTS, and metadata |
5353
| `crdt_budget_fraction` | 0.10 | CRDT state and sync buffers |
54-
| `timeseries_budget_fraction`| 0.10 | Columnar timeseries profile |
54+
| `timeseries_budget_fraction`| 0.10 | Timeseries peer engine |
5555
| `query_budget_fraction` | 0.20 | Query execution working memory |

docs/sql/vector-search.rdx

Lines changed: 72 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -60,17 +60,81 @@ FROM articles
6060
ORDER BY dist LIMIT 10;
6161
```
6262

63-
| Argument | Type | Notes |
64-
| ------------------- | ------- | -------------------------------------------------- |
65-
| `quantization` | string | `none`, `sq8`, `pq`, `ivf_pq`, `rabitq` |
66-
| `oversample` | u32 | Candidates fetched before re-ranking |
67-
| `query_dim` | u32 | Override query dimensionality |
68-
| `meta_token_budget` | u32 | Metadata pre-filter budget |
69-
| `ef_search` | u32 | HNSW search-time `ef` |
70-
| `target_recall` | f32 | Adaptive recall target |
63+
| Argument | Type | Notes |
64+
| ------------------- | ------- | ------------------------------------------------------------------------------------- |
65+
| `quantization` | string | `none`, `sq8`, `pq`, `binary`, `ternary`, `rabitq`, `bbq`, `opq` |
66+
| `oversample` | u8 | Candidates fetched before re-ranking. Default `3`. Final rerank set is `oversample × ef_search`. |
67+
| `query_dim` | u32 | Coarse-to-fine on first-N dims of Matryoshka embeddings. `None` = full dimensionality. |
68+
| `meta_token_budget` | u8 | MetaEmbed multivec scoring budget for MaxSim / PLAID |
69+
| `ef_search` | u32 | HNSW / Vamana search-time beam width. Default `64`. |
70+
| `target_recall` | f32 | Adaptive recall target. The cost-model planner picks oversample / ef_search to hit this. |
7171

7272
Unknown names, duplicate keys, positional 3rd args, or wrong operators (`=` instead of `=>`) all return typed errors that list the canonical names.
7373

74+
### Quantization Choice
75+
76+
| Codec | Bits/dim | Recall (typ.) | When to pick it |
77+
| --------- | -------- | ------------- | -------------------------------------------------------------- |
78+
| `none` | 32 | 100% | Small index (< 1M vectors), latency not critical |
79+
| `sq8` | 8 | ~99% | Balanced default for medium index sizes |
80+
| `pq` | ~2 | ~95% | Large memory-bound indexes; classic Product Quantization |
81+
| `opq` | ~2 | ~96% | PQ + learned random rotation; minor accuracy bump over `pq` |
82+
| `rabitq` | 1 | ~97% | Frontier 1-bit with `O(1/√D)` error bound (SIGMOD 2024) |
83+
| `bbq` | 1 | ~98% | Centroid-asymmetric 1-bit + 14-byte corrective; oversample-rerank |
84+
| `binary` | 1 | ~85% | Hamming-only, no rerank — for ultra-cold tiers |
85+
| `ternary` | 1.58 | ~96% | BitNet-style `{-1, 0, +1}` — cold/hot pack for AVX-512 |
86+
87+
The cost-model planner (`target_recall`) will pick `oversample` and `ef_search` automatically once you set the recall target. Manually set those two only when you need a hard latency ceiling.
88+
89+
## Index Implementations
90+
91+
You don't pick the underlying index directly — it's chosen by the planner from collection metadata + workload signals:
92+
93+
- **HNSW** — in-memory hierarchical graph, the default for moderate-size indexes
94+
- **Vamana / DiskANN** — flat-beam SSD-resident graph for billion-scale on a single node (`Tier 2` of the vector frontier)
95+
- **NaviX adaptive-local filtered traversal** (VLDB 2025) — switches per-hop between standard / directed / blind heuristics based on local selectivity. Replaces classic ACORN-1 filtered ANN.
96+
- **SIEVE workload-driven subindex collections** — the planner builds specialized HNSW subindices for stable predicates (e.g. `tenant_id`) and routes filtered queries to them.
97+
- **MetaEmbed multi-vector + ColBERT MaxSim + PLAID** (ICLR 2026) — learnable Meta Tokens replace per-token explosion; budgeted MaxSim at query time via `meta_token_budget`.
98+
- **Matryoshka adaptive-dim querying** — coarse-to-fine ranking on the first-N dimensions of MRL embeddings via `query_dim`.
99+
- **SPFresh streaming updates** (SOSP 2023) — LIRE topology-aware local rebalancing; no full-rebuild stalls when vectors are added/removed.
100+
101+
## Vector-Primary Collections
102+
103+
By default, vectors are an *index* attached to a column on a normal collection — the document/strict store is the source of truth, and the vector index is a side path. For pure-vector workloads (RAG corpora, recommendation memory, embedding stores) you can flip a collection into **vector-primary** mode, where the vector index becomes the primary access path and the document store is a metadata sidecar:
104+
105+
```sql
106+
CREATE COLLECTION corpus (
107+
id UUID DEFAULT gen_uuid_v7(),
108+
embedding FLOAT[384],
109+
title TEXT,
110+
tenant_id UUID,
111+
created_at TIMESTAMP DEFAULT now()
112+
) WITH (
113+
primary='vector',
114+
vector_field='embedding',
115+
dim=384,
116+
metric='cosine',
117+
quantization='rabitq',
118+
m=32,
119+
ef_construction=200,
120+
payload_indexes=['tenant_id', 'created_at']
121+
);
122+
```
123+
124+
| Option | Notes |
125+
| ----------------- | -------------------------------------------------------------------- |
126+
| `primary` | `'document'` (default), `'strict'`, `'kv'`, or `'vector'` |
127+
| `vector_field` | Required when `primary='vector'`. Column name of the embedding. |
128+
| `dim` | Required. Embedding dimensionality. |
129+
| `metric` | `'cosine'`, `'l2'`, `'inner_product'`, etc. |
130+
| `quantization` | Storage-level codec (same vocabulary as the query-time arg above). |
131+
| `m`, `ef_construction` | HNSW build-time parameters. |
132+
| `payload_indexes` | Per-field equality / range / boolean indexes over the metadata sidecar for filtered ANN. Replaces Pinecone metadata filters. |
133+
134+
In vector-primary mode the planner treats the vector index as the source of truth for IDs; metadata fetches only happen for hit IDs. Cross-engine queries, CRDT sync, and SQL semantics all keep working — `primary='vector'` is purely an *access-path* hint, not a different engine.
135+
136+
Default `primary='document'` is unchanged: the existing `CREATE VECTOR INDEX ON ...` syntax continues to work for vector-as-side-index workloads.
137+
74138
## Hybrid Vector + Text (RRF)
75139

76140
```sql

0 commit comments

Comments
 (0)