Skip to content

Commit bbae449

Browse files
committed
docs: replace all stub pages with complete production-quality content
All 134 stub .rdx files have been fully authored. Coverage spans every documented section: - Getting Started: installation, Docker, quickstart, server config, engine tuning, memory budgets - Concepts: hybrid execution model, control/data plane separation, SPSC bridge, WAL, storage tiers, CRDT, vShards, Raft, replication, shard migration, consistency, snapshot isolation, compaction, segments, schemas, tenants, data types, write path - Engines: vector (HNSW, quantization, distance metrics, pre-filtering, indexing params), CRDT (merge semantics, conflict resolution, constraint validation, dead-letter queue), sparse/relational (B-tree indexes, BM25, range queries, schema management), timeseries (compression, retention, metrics and log ingestion), engine budgets, engine isolation, fusion search - Query: SQL overview, SELECT, DDL, INSERT/UPSERT, UPDATE/DELETE, vector search, filtered search, multi-vector, RRF fusion, cross-model queries, CRDT apply/policy/read, change streams, COPY protocol, prepared statements, EXPLAIN, functions, graph-RAG - Drivers: pgwire (compatibility, session variables, supported features, type mapping), HTTP API (query endpoint, document API, search API, health/metrics), psql, Python driver, Rust client, ORMs, connection poolers, DBeaver/pgAdmin - Operations: cluster setup, Docker deployment, Kubernetes, systemd, authentication, TLS, RBAC, tenant isolation, health checks, key metrics, Prometheus, OpenTelemetry, slow query log, audit logging, memory management, WAL archiving, online backup, PITR, restore, rolling upgrades, node management, shard rebalancing, failure recovery, benchmarking, production checklist, tuning guide, io_uring tuning - Reference: server.toml, engine config, cluster config, defaults, limits, feature flags, error codes, SQLSTATE mapping, type OIDs, pgwire extensions, HTTP API spec, native protocol spec, WAL format, segment format, SPSC envelope, bridge errors Broken /docs/ prefix links fixed throughout. Server config values validated and corrected against the actual codebase.
1 parent 4a69908 commit bbae449

134 files changed

Lines changed: 38648 additions & 335 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

concepts/cluster-topology.rdx

Lines changed: 383 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,389 @@
11
---
22
title: Cluster Topology
3-
description: Node roles, join/leave/decommission lifecycle.
3+
description: Cluster architecture, node roles (leader, follower, learner), bootstrap, join, leave, and decommission operations.
44
---
55

66
# Cluster Topology
77

8-
TODO
8+
A NodeDB cluster is a group of nodes running the hybrid execution model together, replicating data via Multi-Raft. The cluster must always have an odd number of nodes (3, 5, 7, etc.) for quorum consensus.
9+
10+
## Node Roles
11+
12+
Every node in the cluster has a role determined by the Raft consensus layer:
13+
14+
### Leader Node
15+
16+
**Responsibilities:**
17+
- Accepts writes from clients
18+
- Replicates writes to followers
19+
- Maintains leadership (heartbeat every 150 ms)
20+
- Resolves conflicts in case of simultaneous writes
21+
22+
**Behavior:**
23+
```
24+
Client sends write
25+
26+
Leader appends to WAL
27+
28+
Leader sends AppendEntries RPC to followers
29+
30+
Followers acknowledge
31+
32+
Leader commits (quorum)
33+
34+
Leader applies to L0
35+
36+
Leader sends ACK to client
37+
```
38+
39+
**Election:**
40+
- Maintains position by sending heartbeats
41+
- If followers don't receive heartbeat for 300 ms, they trigger election
42+
- Candidate with majority votes becomes leader
43+
44+
### Follower Node
45+
46+
**Responsibilities:**
47+
- Accept read requests (if consistency allows)
48+
- Receive replication from leader
49+
- Vote in leader elections
50+
- Apply committed entries to L0
51+
52+
**Behavior:**
53+
```
54+
Receive write request
55+
56+
IF CONSISTENCY='STRONG':
57+
→ Forward to leader
58+
ELSE IF CONSISTENCY='BOUNDED_STALENESS(1s)':
59+
→ Check replication lag
60+
→ If lag <= 1s: execute locally
61+
→ Else: forward to leader
62+
63+
Receive AppendEntries from leader
64+
→ Append to WAL
65+
→ Send ACK
66+
→ Apply when leader says commit_index
67+
68+
Receive heartbeat (empty AppendEntries)
69+
→ Update election timer
70+
→ Stay follower
71+
```
72+
73+
### Learner Node
74+
75+
**Responsibilities:**
76+
- Participate in replication (receive writes)
77+
- Do NOT vote in elections
78+
- Do NOT serve reads
79+
- Used for safe cluster expansion
80+
81+
**Behavior:**
82+
```
83+
Learner is promoted to follower:
84+
1. Learner joins cluster with no voting rights
85+
2. Replicates data from leader
86+
3. Once caught up (lag < 1s): promote to voter
87+
4. Learner becomes full follower
88+
89+
Use case: Adding node 4 to 3-node cluster
90+
Before: 3 nodes, quorum = 2
91+
Add node 4 as learner: 3 voters + 1 learner, quorum still = 2
92+
Node 4 catches up: 4 voters, quorum = 3
93+
No disruption during catch-up
94+
```
95+
96+
## Cluster Modes
97+
98+
### Single-Node
99+
100+
Useful for embedded, development, and single-machine deployments.
101+
102+
```
103+
nodedb --single-node --data-dir ./data
104+
105+
- 1 node is always leader
106+
- No replication
107+
- No failover
108+
- WAL is still persisted (crash recovery works)
109+
```
110+
111+
### Quorum (3 nodes)
112+
113+
Minimum production setup. Can tolerate 1 node failure.
114+
115+
```
116+
nodedb --cluster-name "prod"
117+
--seed-nodes "node1:6379,node2:6379,node3:6379"
118+
119+
Cluster:
120+
node1 (leader) → replicates to node2, node3
121+
node2 (follower)
122+
node3 (follower)
123+
124+
Quorum: 2 out of 3
125+
- If node1 dies: node2/node3 elect new leader, cluster continues
126+
- If any 1 node dies: quorum (2/3) survives
127+
- If 2 nodes die: quorum lost, cluster read-only
128+
```
129+
130+
### Larger Clusters (5, 7, 9 nodes)
131+
132+
Used for higher availability and geographically distributed systems.
133+
134+
```
135+
5-node cluster:
136+
Quorum = 3
137+
Can tolerate 2 node failures
138+
139+
7-node cluster:
140+
Quorum = 4
141+
Can tolerate 3 node failures
142+
143+
Distributed example:
144+
US-EAST: node1, node2 (local quorum: 2 nodes)
145+
US-WEST: node3, node4 (local quorum: 2 nodes)
146+
EU: node5 (tiebreaker)
147+
148+
Latency:
149+
US-EAST local writes: ~1 ms
150+
US-EAST to US-WEST: ~50 ms
151+
US to EU: ~100 ms
152+
```
153+
154+
## Bootstrap Process
155+
156+
Creating a new cluster:
157+
158+
```
159+
1. Start 3 nodes with cluster config
160+
node1 --cluster-name "prod" --node-id 1 --bind 0.0.0.0:5433
161+
node2 --cluster-name "prod" --node-id 2 --bind 0.0.0.0:5433
162+
node3 --cluster-name "prod" --node-id 3 --bind 0.0.0.0:5433
163+
164+
2. Bootstrap cluster (on any node)
165+
nodedb-cli --node=node1 cluster bootstrap \
166+
--members "1@node1:6379,2@node2:6379,3@node3:6379"
167+
168+
3. Nodes exchange heartbeats
169+
Leader election happens automatically
170+
One of 3 becomes leader
171+
172+
4. Cluster is READY
173+
Accepts reads and writes
174+
```
175+
176+
Or use pre-configured seed nodes:
177+
178+
```
179+
All nodes configured with:
180+
--seed-nodes "node1:6379,node2:6379,node3:6379"
181+
182+
Nodes auto-discover and form cluster (no explicit bootstrap).
183+
```
184+
185+
## Join a Node
186+
187+
Adding a node to an existing cluster:
188+
189+
```
190+
Existing 3-node cluster (node1, node2, node3)
191+
Want to add node4 to make 5-node cluster
192+
193+
Steps:
194+
195+
1. Start new node as learner
196+
node4 --cluster-name "prod" --node-id 4
197+
--learner
198+
--seed-nodes "node1:6379,node2:6379"
199+
200+
2. node4 joins, receives replication
201+
node4 is learner (doesn't vote)
202+
Quorum remains 2 (only node1, node2, node3 vote)
203+
204+
3. Wait for node4 to catch up
205+
Monitor: nodedb-cli cluster status
206+
node4 replication_lag < 100 ms
207+
208+
4. Promote learner to voter
209+
nodedb-cli --node=node1 cluster add-voter --node-id 4
210+
211+
5. node4 is now full member
212+
New quorum = 3 (out of 5)
213+
Cluster can tolerate 2 failures
214+
```
215+
216+
## Leave Cluster
217+
218+
Gracefully remove a node:
219+
220+
```
221+
Existing 3-node cluster
222+
Want to remove node3
223+
224+
Steps:
225+
226+
1. Initiate graceful shutdown on node3
227+
nodedb-cli --node=node3 node shutdown --graceful
228+
229+
2. node3 stops accepting new requests
230+
Drains in-flight requests
231+
232+
3. node3 notifies leader: "I'm leaving"
233+
Leader issues ConfChange: REMOVE_VOTER(node3)
234+
Leader waits for quorum ACK
235+
236+
4. Leader removes node3 from quorum
237+
New quorum = 2 (node1, node2)
238+
Followers apply ConfChange
239+
240+
5. node3 exits
241+
Cluster now 2 nodes (unstable quorum)
242+
243+
6. (Recommended) Add new node4 to restore quorum
244+
Otherwise cluster is vulnerable (any failure = partition)
245+
```
246+
247+
## Decommission
248+
249+
Permanently retire a node (e.g., hardware replacement):
250+
251+
```
252+
Steps:
253+
254+
1. Drain data: Shard migration
255+
nodedb-cli cluster drain-node --node-id 3
256+
257+
For each vShard owned by node3:
258+
- Identify target replica node (node1 or node2)
259+
- Copy data (base + WAL catch-up + atomic cut-over)
260+
- Update routing table
261+
- Data now on target node
262+
263+
2. Remove from cluster
264+
nodedb-cli cluster remove-voter --node-id 3
265+
266+
3. node3 is no longer part of cluster
267+
Can be powered off
268+
269+
4. New node can be added later (if needed)
270+
nodedb-cli cluster add-learner --node-id 4
271+
```
272+
273+
## Failure Scenarios
274+
275+
### Single Node Failure
276+
277+
```
278+
3-node cluster (node1 is leader):
279+
Quorum = 2
280+
281+
node1 crashes
282+
├─ node1 stops sending heartbeats
283+
├─ node2, node3 don't receive heartbeat for 300 ms
284+
├─ node2, node3 timeout, trigger election
285+
├─ One of (node2, node3) wins election (2 votes = quorum)
286+
├─ New leader is elected
287+
└─ Cluster continues (reads and writes work)
288+
289+
Recovery: node1 rejoins
290+
├─ node1 boots, reads persistent state
291+
├─ Discovers it was leader but term is old
292+
├─ Applies itself as follower (new leader already exists)
293+
├─ Catches up via replication from new leader
294+
└─ Cluster is back to 3 nodes
295+
```
296+
297+
### Network Partition
298+
299+
```
300+
3-node cluster split:
301+
Side A: node1, node2 (can talk to each other)
302+
Side B: node3 (isolated)
303+
304+
Side A:
305+
├─ node1, node2 have quorum (2/3)
306+
├─ Leader stays active on side A
307+
├─ Accepts reads and writes
308+
309+
Side B:
310+
├─ node3 has no quorum (1/3)
311+
├─ node3 stops accepting writes
312+
├─ Serves reads with EVENTUAL consistency
313+
314+
Reconciliation: partition heals
315+
├─ node3 reconnects
316+
├─ node3 is behind (missed writes on side A)
317+
├─ Followers send missing entries
318+
├─ node3 catches up
319+
└─ Cluster is consistent again (no data loss)
320+
```
321+
322+
### Leader Death + Network Partition
323+
324+
```
325+
5-node cluster (node1 is leader):
326+
Quorum = 3
327+
328+
node1 dies + network partition:
329+
Side A: node2, node3 (can talk)
330+
Side B: node4, node5 (can talk)
331+
Partitioned: A <-> B
332+
333+
Side A (quorum = 2/3, not enough):
334+
├─ node2, node3 cannot elect leader (need 3)
335+
├─ No leader on side A
336+
└─ Reads: allowed, Writes: blocked
337+
338+
Side B (quorum = 2/5, not enough):
339+
├─ node4, node5 cannot elect leader (need 3)
340+
├─ No leader on side B
341+
└─ Reads: allowed, Writes: blocked
342+
343+
Cluster is SPLIT-BRAIN SAFE:
344+
├─ Neither side can write (no quorum)
345+
├─ No conflicting writes possible
346+
├─ When partition heals, one side is source of truth
347+
```
348+
349+
## Monitoring
350+
351+
Check cluster health:
352+
353+
```bash
354+
nodedb-cli cluster status
355+
356+
Output:
357+
cluster_id: "abc123"
358+
node_count: 3
359+
quorum: 2
360+
status: HEALTHY
361+
362+
Members:
363+
node_id: 1
364+
role: LEADER
365+
status: ALIVE
366+
term: 100
367+
commit_index: 50000
368+
369+
node_id: 2
370+
role: FOLLOWER
371+
status: ALIVE
372+
replication_lag: 50 ms (behind leader)
373+
374+
node_id: 3
375+
role: FOLLOWER
376+
status: ALIVE
377+
replication_lag: 30 ms (behind leader)
378+
```
379+
380+
## Performance Characteristics
381+
382+
| Operation | Single-Node | 3-Node Cluster | 5-Node Cluster |
383+
|-----------|------------|----------------|----------------|
384+
| Write latency | 1-5 ms | 5-20 ms | 5-20 ms |
385+
| Strong read | 1-5 ms | 1-5 ms | 1-5 ms |
386+
| Bounded-staleness read | < 1 ms | 1-10 ms | 1-10 ms |
387+
| Failover time | N/A | 300-500 ms | 300-500 ms |
388+
| Network overhead | 0 | ~300 Kbps | ~300 Kbps |
389+
| Max quorum loss tolerance | 0 | 1 failure | 2 failures |

0 commit comments

Comments
 (0)