You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
NodeDB 0.4.0 is a substantial distributed-correctness and durability release. It adds cross-shard transactional execution and distributed query/graph processing, extends temporal and sparse-vector SQL, and hardens replication, recovery, indexing, authentication, and sync across the storage engines.
13
+
12
14
### ⚠️ Breaking changes
13
15
14
16
-**On-disk storage layout is now database-scoped.** Every engine (Document, KV, Columnar, Timeseries, Spatial, Vector, sparse-vector, Graph, FTS) keys its storage maps and on-disk paths by `database_id`, and the PK→surrogate catalog is scoped to `(database_id, tenant_id, collection, pk)`. **A 0.3.0 data directory will not be found by a 0.4.0 binary** — a dump/reload is required to upgrade.
15
-
-**WAL / on-disk format evolved (forward-incompatible).** New *required* WAL record types (`SyncSeqAdvance`, `FtsIndex`, `FtsDelete`, `SpatialPut`, `SpatialDelete`), a map-encoded `ColumnarWalRecord` carrying per-row surrogates, and new replayable KV / graph-label / CRDT-list / columnar-predicate records. A 0.3.0 binary cannot replay a 0.4.0 WAL; a 0.4.0 binary still replays 0.3.0 logs.
17
+
-**WAL / on-disk format evolved (forward-incompatible).** New _required_ WAL record types (`SyncSeqAdvance`, `FtsIndex`, `FtsDelete`, `SpatialPut`, `SpatialDelete`), a map-encoded `ColumnarWalRecord` carrying per-row surrogates, and new replayable KV / graph-label / CRDT-list / columnar-predicate records. A 0.3.0 binary cannot replay a 0.4.0 WAL; a 0.4.0 binary still replays 0.3.0 logs.
16
18
-**`NodeDb` trait** gained CRDT movable-list operations — trait implementors and callers must update.
17
19
-**`single_node_calvin` now defaults to `true`.** A standalone node stands up the single-node Calvin sequencer by default, and cross-core (cross-vShard) transactions commit atomically through it instead of being rejected. Restore the prior behavior with `single_node_calvin = false`.
18
20
-**SQL authorization is now enforced across all transports.** Statements that previously bypassed authorization on the native/HTTP paths are now rejected.
19
21
-**Cluster wire compatibility** — removed superseded iterative-WCC and GraphAlgo-BSP wire types; all cluster nodes must run the same minor version.
22
+
-**The sync wire frame is incompatible with 0.3.x clients.** Sync now uses a versioned frame with CRC32C integrity and idempotent-producer metadata. Upgrade NodeDB Lite and other sync clients together with the server.
23
+
-**Static JWT and catalog OIDC providers must be bound to a server-trusted tenant.** Static `auth.jwt.providers` entries now require `tenant_id`, and catalog providers require `TENANT`; existing catalog provider records without that binding are rejected until recreated. Shared issuers must also use distinct, non-empty audiences so issuer/audience routing is unambiguous.
20
24
21
25
### Added
22
26
23
-
-**Distributed ACID transactions** — a deterministic Calvin sequencer for cross-shard/cross-vShard commits with per-participant vote tally, verdict barrier, and WAL-replayable redo records; a per-transaction staging overlay serving reads and buffering writes for **every** engine (point/predicate/INSERT…SELECT writes, KV atomic/batch/TTL ops, document UPSERT, columnar batch inserts, vector search, spatial predicate scans, graph single-/multi-hop/shortest-path, full-text search); full undo/rollback with index, R-tree, column-statistics, and bitemporal-aware reversal; `SAVEPOINT` / `RELEASE` / `ROLLBACK TO SAVEPOINT` on both pgwire and the native protocol; wound-wait lock arbitration, per-vShard deterministic write fencing, and a write-admission gate at the SPSC chokepoint.
27
+
-**Distributed ACID transactions** — a deterministic Calvin sequencer for cross-shard/cross-vShard commits with per-participant vote tally, verdict barrier, failover recovery, serialization-failure reporting, and WAL-replayable redo records; a per-transaction staging overlay serving reads and buffering writes for **every** engine (point/predicate/`INSERT … SELECT` writes, KV atomic/batch/TTL ops, document UPSERT, columnar batch inserts, vector search, spatial predicate scans, graph single-/multi-hop/shortest-path, full-text search); full undo/rollback with index, R-tree, column-statistics, and bitemporal-aware reversal; wound-wait lock arbitration, per-vShard deterministic write fencing, and a write-admission gate at the SPSC chokepoint.
28
+
-**Complete transaction and DML lifecycle across client protocols** — native sessions now support transaction setup and teardown plus `SAVEPOINT` / `RELEASE` / `ROLLBACK TO SAVEPOINT`, matching pgwire behavior; transactional `MERGE`, `UPDATE … FROM`, and `INSERT … SELECT` resolve dependent reads into point operations while preserving overlays, `RETURNING`, rollback, replay, and cross-shard commit semantics. Abandoned pgwire and native sessions reclaim their transaction overlays.
24
29
-**MVCC read validation for distributed OCC** — LSN-versioned read-sets validated against committed write-versions at apply time; per-collection and per-index (`IndexEq`/`IndexRange`) read-version tracking; cross-shard OCC self-abort and read-only-participant handling for shuffle/gather/gathered JOINs.
25
30
-**Distributed query execution** — cross-node streaming `Exchange` / `ProviderScan` over QUIC; distributed shuffle JOIN and shuffle GROUP BY with ANALYZE-driven cost-model auto-selection; grace-hash join with recursive re-partitioning and io_uring spill-to-disk; memory-budget-bounded streaming scans replacing silent row caps; lazy streaming of unordered `SELECT`s over pgwire, native, HTTP, and QUIC.
26
31
-**Distributed graph** — cross-shard `MATCH` with multi-round continuation and variable-length resume cursors; distributed BSP PageRank, Personalized PageRank, and WCC; dual-homed cross-shard edge insert/delete routed through Calvin with implicit-edge reconciliation on predicate UPDATE/DELETE; owner-partitioned BFS frontier for full cross-node traversal.
27
32
-**CRDT via SQL** — `WITH (crdt=true)` on document collections routes DML to CRDT ops with `RETURNING` support; movable-list ops over the native wire protocol and `NodeDb` trait; committed-delta validation at data-plane apply time (CHECK constraints, write-set extraction, descriptor-version fencing); validator rejections surfaced as `DeltaReject`; CRDT writes made quorum-durable under Raft.
28
33
-**Sparse-vector engine** — a dimensionless `SPARSEVECTOR` column type threaded through parser, wire format, columnar/strict storage, and DDL; an inverted index maintained on document writes; a `sparse_score``ORDER BY` surface for sparse-vector search.
29
-
-**Cluster elasticity & membership** — rendezvous-hashing placement with explicit per-group placement sets; learner add/remove/auto-promotion/eviction; Raft leadership transfer via `TimeoutNow`; immediate placement reconcile on node join; leaving-voter convergence; orphan partial-snapshot GC.
34
+
-**Cluster elasticity & membership** — rendezvous-hashing placement with explicit per-group placement sets; learner add/remove/auto-promotion/eviction; Raft leadership transfer via `TimeoutNow`; immediate placement reconcile on node join; leaving-voter convergence; orphan partial-snapshot GC; and HiLo batch allocation for cross-node row surrogates.
30
35
-**Raft snapshots & compaction** — a Data-Plane snapshot builder wired into the Raft SEND path; exact clear-then-install for lagging followers; CRDT state, graph edges, columnar/timeseries engine state, and the PK→surrogate identity map carried through per-group snapshots and backups; configurable Raft log auto-compaction gated on the durable Data-Plane applied watermark.
31
36
-**Idempotent sync** — idempotent-producer wire types with frame integrity; a per-core idempotency gate for sync ingest; sync-HWM replay on startup; Raft-replicated sync writes for FTS, spatial, columnar, timeseries, and vector; collection-schema announcement and peer-collection materialization into the local catalog.
32
-
-**Bitemporal** — a `SystemTimeScope` threaded through planner, cluster executor, clone rewriter, and the array/document/columnar/graph dispatchers; audit-log scans over the array and sparse-document stores.
37
+
-**Bitemporal SQL and execution** — `FOR SYSTEM_TIME AS OF` and valid-time qualifiers are carried through planning and distributed execution for supported engines. `AS OF SYSTEM TIME NULL` exposes complete version history with temporal bounds for strict and schemaless documents, columnar collections, and timeseries collections, while reserved temporal columns stay out of ordinary projections.
33
38
-**pgwire / SQL surface** — `version()`, `current_setting()`, `server_version_num` in startup params, and an extended `pg_catalog` evaluator for driver compatibility; `ST_MakePoint` / `ST_GeomFromText` / `ST_GeomFromWKB` in `INSERT` values; computed expressions in `GROUP BY` keys; a MySQL-style trailing `ENGINE = <name>` clause on `CREATE COLLECTION`; per-collection FTS analyzer honored in every tokenization path.
34
39
-**`RESTORE TENANT ... FORCE`** — bypass the staleness guard on restore.
35
40
36
41
### Changed
37
42
38
43
-**Response shaping is now planner-driven** — a protocol-neutral output-schema builder wired into query planning drives response shaping and `Describe` across pgwire, native, and HTTP, replacing the per-transport projection paths.
39
44
-**CDC change events are published exactly once per write**, and graph node-label/edge writes and cluster-array writes now emit CDC events; the Event-Plane receiver is wired into vShard dispatch.
40
-
-**Native autocommit writes route through Raft**, and the durable-at-ack barrier extends to pgwire submit and ILP.
45
+
-**Replication and acknowledgement durability** — native autocommit writes route through Raft, the durable-at-ack barrier extends to pgwire submit and ILP, and async WAL group commit does not acknowledge before the durability barrier. Previously uncovered array-cell, bulk document, KV truncate, CRDT, vector, sparse-vector, graph-label, and graph-index write paths now replicate through Raft or WAL as appropriate. Startup replay and Raft compaction are gated on the durable Data-Plane applied watermark, while dropped Calvin fan-out is recovered from the sequencer log.
46
+
-**Cross-engine identity and index maintenance** — bulk KV/native writes, generated-key inserts, columnar batches, and replicated writes now preserve or mint stable surrogates consistently. Index reconciliation was hardened across bulk DML, `MERGE`, `UPDATE … FROM`, rollback, `TRUNCATE`, restore, and WAL replay, including secondary-field, vector, FTS, and spatial index paths.
41
47
-**License** — `nodedb-crdt`, `nodedb-mem`, and `nodedb-wal` relicensed to Apache-2.0.
42
48
43
49
### Fixed
44
50
45
51
-**Crash-safety hardening** — boot now fails on a corrupt/unreadable checkpoint for every engine (vector, spatial, graph-label, columnar, KV, sparse-vector, timeseries, CRDT, sync-HWM) instead of silently continuing; CRC framing added to checkpoint files; checkpoint encode/restore failures propagated instead of swallowed.
46
52
-**WAL replay correctness** — timeseries samples are no longer rejected during replay and a mid-record flush no longer duplicates rows on recovery; per-row surrogates are persisted and restored through columnar WAL replay; complete KV WAL replay (`incr`/`expire`/`persist`/`cas`/`field_set`/`register_index`/`drop_index`) with resolved expiry instants; graph node-label and columnar predicate UPDATE/DELETE records persisted and replayed.
53
+
-**Query and SQL correctness** — scan and join execution no longer silently truncates rows at implicit caps; streamed scans drain every chunk; bitemporal range scans, indexed residual predicates, computed and distributed `GROUP BY`, and scalar aggregates over empty input return complete results; strict `AS OF SYSTEM TIME NULL` queries resolve the correct historical schema.
47
54
-**Restore / backup** — WAL tombstones replicated via Raft on restore; plain-columnar rows re-issued durably rather than snapshot-installed; columnar/flushed-timeseries data and catalog propagated cluster-wide; replica multiplication and CRDT loss under RF>1 prevented.
48
55
-**Tenant management** — tenant IDs allocated via a durable high-water-mark; ghost rows in `SHOW TENANTS` after `DROP TENANT` eliminated; an existence gate enforced for unknown numeric tenant IDs; `DROP`/`ALTER`/`PURGE TENANT` accept a tenant name.
49
56
-**Security** — UTF-8 SQL parser offsets preserved; external superuser assertions prevented; OIDC providers bound to trusted tenants; credential integrity preserved across the auth lifecycle; legitimate login bursts no longer trip rate limits.
50
57
-**KV** — TTL-expired rows reaped without stranding index entries; per-row surrogates assigned on native/RESP batch-put and MSET so cross-engine joins over bulk KV writes resolve.
51
58
-**Cluster** — remote error codes preserved across RPC; `WrongOwner` excluded from circuit-breaker failures; Raft compaction and restart replay gated on durable apply; AFTER-trigger writes routed to the owning shard; SQL-inserted geometry indexed into the R-tree on document collections.
59
+
-**Graph and CRDT correctness** — distributed PageRank accounts for dangling rank mass across shards; variable-length `MATCH` resumes without dropping frontier work or duplicating seeds; cross-shard graph mutations reconcile edge ownership and labels; CRDT state is isolated per collection, quorum-replicated, snapshot-safe, materialized into document storage, and rejects malformed or constraint-violating deltas before apply.
52
60
-**Vector** — the prior HNSW node is removed before re-insert on a vector put.
53
61
62
+
### Quality
63
+
64
+
-**Multi-node and crash-recovery coverage** — expanded coverage for Calvin/OCC, failover, shuffle queries, snapshots, placement, sync, graph, and CRDT behavior; added real process-kill, WAL-truncation, checkpoint-corruption, and repeated-restart scenarios; hardened CI disk management, dependency policy, and concurrent server-test isolation.
65
+
54
66
### Removed
55
67
56
68
- Dead pgwire projection, catalog-propose, and DDL-router modules (superseded by the neutral shaping/DDL router).
0 commit comments