|
1 | 1 | --- |
2 | 2 | title: Cluster Topology |
3 | | -description: Node roles, join/leave/decommission lifecycle. |
| 3 | +description: Cluster architecture, node roles (leader, follower, learner), bootstrap, join, leave, and decommission operations. |
4 | 4 | --- |
5 | 5 |
|
6 | 6 | # Cluster Topology |
7 | 7 |
|
8 | | -TODO |
| 8 | +A NodeDB cluster is a group of nodes running the hybrid execution model together, replicating data via Multi-Raft. The cluster must always have an odd number of nodes (3, 5, 7, etc.) for quorum consensus. |
| 9 | + |
| 10 | +## Node Roles |
| 11 | + |
| 12 | +Every node in the cluster has a role determined by the Raft consensus layer: |
| 13 | + |
| 14 | +### Leader Node |
| 15 | + |
| 16 | +**Responsibilities:** |
| 17 | +- Accepts writes from clients |
| 18 | +- Replicates writes to followers |
| 19 | +- Maintains leadership (heartbeat every 150 ms) |
| 20 | +- Resolves conflicts in case of simultaneous writes |
| 21 | + |
| 22 | +**Behavior:** |
| 23 | +``` |
| 24 | +Client sends write |
| 25 | + ↓ |
| 26 | +Leader appends to WAL |
| 27 | + ↓ |
| 28 | +Leader sends AppendEntries RPC to followers |
| 29 | + ↓ |
| 30 | +Followers acknowledge |
| 31 | + ↓ |
| 32 | +Leader commits (quorum) |
| 33 | + ↓ |
| 34 | +Leader applies to L0 |
| 35 | + ↓ |
| 36 | +Leader sends ACK to client |
| 37 | +``` |
| 38 | + |
| 39 | +**Election:** |
| 40 | +- Maintains position by sending heartbeats |
| 41 | +- If followers don't receive heartbeat for 300 ms, they trigger election |
| 42 | +- Candidate with majority votes becomes leader |
| 43 | + |
| 44 | +### Follower Node |
| 45 | + |
| 46 | +**Responsibilities:** |
| 47 | +- Accept read requests (if consistency allows) |
| 48 | +- Receive replication from leader |
| 49 | +- Vote in leader elections |
| 50 | +- Apply committed entries to L0 |
| 51 | + |
| 52 | +**Behavior:** |
| 53 | +``` |
| 54 | +Receive write request |
| 55 | + ↓ |
| 56 | +IF CONSISTENCY='STRONG': |
| 57 | + → Forward to leader |
| 58 | +ELSE IF CONSISTENCY='BOUNDED_STALENESS(1s)': |
| 59 | + → Check replication lag |
| 60 | + → If lag <= 1s: execute locally |
| 61 | + → Else: forward to leader |
| 62 | + |
| 63 | +Receive AppendEntries from leader |
| 64 | + → Append to WAL |
| 65 | + → Send ACK |
| 66 | + → Apply when leader says commit_index |
| 67 | + |
| 68 | +Receive heartbeat (empty AppendEntries) |
| 69 | + → Update election timer |
| 70 | + → Stay follower |
| 71 | +``` |
| 72 | + |
| 73 | +### Learner Node |
| 74 | + |
| 75 | +**Responsibilities:** |
| 76 | +- Participate in replication (receive writes) |
| 77 | +- Do NOT vote in elections |
| 78 | +- Do NOT serve reads |
| 79 | +- Used for safe cluster expansion |
| 80 | + |
| 81 | +**Behavior:** |
| 82 | +``` |
| 83 | +Learner is promoted to follower: |
| 84 | + 1. Learner joins cluster with no voting rights |
| 85 | + 2. Replicates data from leader |
| 86 | + 3. Once caught up (lag < 1s): promote to voter |
| 87 | + 4. Learner becomes full follower |
| 88 | + |
| 89 | +Use case: Adding node 4 to 3-node cluster |
| 90 | + Before: 3 nodes, quorum = 2 |
| 91 | + Add node 4 as learner: 3 voters + 1 learner, quorum still = 2 |
| 92 | + Node 4 catches up: 4 voters, quorum = 3 |
| 93 | + No disruption during catch-up |
| 94 | +``` |
| 95 | + |
| 96 | +## Cluster Modes |
| 97 | + |
| 98 | +### Single-Node |
| 99 | + |
| 100 | +Useful for embedded, development, and single-machine deployments. |
| 101 | + |
| 102 | +``` |
| 103 | +nodedb --single-node --data-dir ./data |
| 104 | + |
| 105 | +- 1 node is always leader |
| 106 | +- No replication |
| 107 | +- No failover |
| 108 | +- WAL is still persisted (crash recovery works) |
| 109 | +``` |
| 110 | + |
| 111 | +### Quorum (3 nodes) |
| 112 | + |
| 113 | +Minimum production setup. Can tolerate 1 node failure. |
| 114 | + |
| 115 | +``` |
| 116 | +nodedb --cluster-name "prod" |
| 117 | + --seed-nodes "node1:6379,node2:6379,node3:6379" |
| 118 | + |
| 119 | +Cluster: |
| 120 | + node1 (leader) → replicates to node2, node3 |
| 121 | + node2 (follower) |
| 122 | + node3 (follower) |
| 123 | + |
| 124 | +Quorum: 2 out of 3 |
| 125 | +- If node1 dies: node2/node3 elect new leader, cluster continues |
| 126 | +- If any 1 node dies: quorum (2/3) survives |
| 127 | +- If 2 nodes die: quorum lost, cluster read-only |
| 128 | +``` |
| 129 | + |
| 130 | +### Larger Clusters (5, 7, 9 nodes) |
| 131 | + |
| 132 | +Used for higher availability and geographically distributed systems. |
| 133 | + |
| 134 | +``` |
| 135 | +5-node cluster: |
| 136 | + Quorum = 3 |
| 137 | + Can tolerate 2 node failures |
| 138 | + |
| 139 | +7-node cluster: |
| 140 | + Quorum = 4 |
| 141 | + Can tolerate 3 node failures |
| 142 | + |
| 143 | +Distributed example: |
| 144 | + US-EAST: node1, node2 (local quorum: 2 nodes) |
| 145 | + US-WEST: node3, node4 (local quorum: 2 nodes) |
| 146 | + EU: node5 (tiebreaker) |
| 147 | + |
| 148 | +Latency: |
| 149 | + US-EAST local writes: ~1 ms |
| 150 | + US-EAST to US-WEST: ~50 ms |
| 151 | + US to EU: ~100 ms |
| 152 | +``` |
| 153 | + |
| 154 | +## Bootstrap Process |
| 155 | + |
| 156 | +Creating a new cluster: |
| 157 | + |
| 158 | +``` |
| 159 | +1. Start 3 nodes with cluster config |
| 160 | + node1 --cluster-name "prod" --node-id 1 --bind 0.0.0.0:5433 |
| 161 | + node2 --cluster-name "prod" --node-id 2 --bind 0.0.0.0:5433 |
| 162 | + node3 --cluster-name "prod" --node-id 3 --bind 0.0.0.0:5433 |
| 163 | + |
| 164 | +2. Bootstrap cluster (on any node) |
| 165 | + nodedb-cli --node=node1 cluster bootstrap \ |
| 166 | + --members "1@node1:6379,2@node2:6379,3@node3:6379" |
| 167 | + |
| 168 | +3. Nodes exchange heartbeats |
| 169 | + Leader election happens automatically |
| 170 | + One of 3 becomes leader |
| 171 | + |
| 172 | +4. Cluster is READY |
| 173 | + Accepts reads and writes |
| 174 | +``` |
| 175 | + |
| 176 | +Or use pre-configured seed nodes: |
| 177 | + |
| 178 | +``` |
| 179 | +All nodes configured with: |
| 180 | + --seed-nodes "node1:6379,node2:6379,node3:6379" |
| 181 | + |
| 182 | +Nodes auto-discover and form cluster (no explicit bootstrap). |
| 183 | +``` |
| 184 | + |
| 185 | +## Join a Node |
| 186 | + |
| 187 | +Adding a node to an existing cluster: |
| 188 | + |
| 189 | +``` |
| 190 | +Existing 3-node cluster (node1, node2, node3) |
| 191 | +Want to add node4 to make 5-node cluster |
| 192 | + |
| 193 | +Steps: |
| 194 | + |
| 195 | +1. Start new node as learner |
| 196 | + node4 --cluster-name "prod" --node-id 4 |
| 197 | + --learner |
| 198 | + --seed-nodes "node1:6379,node2:6379" |
| 199 | + |
| 200 | +2. node4 joins, receives replication |
| 201 | + node4 is learner (doesn't vote) |
| 202 | + Quorum remains 2 (only node1, node2, node3 vote) |
| 203 | + |
| 204 | +3. Wait for node4 to catch up |
| 205 | + Monitor: nodedb-cli cluster status |
| 206 | + node4 replication_lag < 100 ms |
| 207 | + |
| 208 | +4. Promote learner to voter |
| 209 | + nodedb-cli --node=node1 cluster add-voter --node-id 4 |
| 210 | + |
| 211 | +5. node4 is now full member |
| 212 | + New quorum = 3 (out of 5) |
| 213 | + Cluster can tolerate 2 failures |
| 214 | +``` |
| 215 | + |
| 216 | +## Leave Cluster |
| 217 | + |
| 218 | +Gracefully remove a node: |
| 219 | + |
| 220 | +``` |
| 221 | +Existing 3-node cluster |
| 222 | +Want to remove node3 |
| 223 | + |
| 224 | +Steps: |
| 225 | + |
| 226 | +1. Initiate graceful shutdown on node3 |
| 227 | + nodedb-cli --node=node3 node shutdown --graceful |
| 228 | + |
| 229 | +2. node3 stops accepting new requests |
| 230 | + Drains in-flight requests |
| 231 | + |
| 232 | +3. node3 notifies leader: "I'm leaving" |
| 233 | + Leader issues ConfChange: REMOVE_VOTER(node3) |
| 234 | + Leader waits for quorum ACK |
| 235 | + |
| 236 | +4. Leader removes node3 from quorum |
| 237 | + New quorum = 2 (node1, node2) |
| 238 | + Followers apply ConfChange |
| 239 | + |
| 240 | +5. node3 exits |
| 241 | + Cluster now 2 nodes (unstable quorum) |
| 242 | + |
| 243 | +6. (Recommended) Add new node4 to restore quorum |
| 244 | + Otherwise cluster is vulnerable (any failure = partition) |
| 245 | +``` |
| 246 | + |
| 247 | +## Decommission |
| 248 | + |
| 249 | +Permanently retire a node (e.g., hardware replacement): |
| 250 | + |
| 251 | +``` |
| 252 | +Steps: |
| 253 | + |
| 254 | +1. Drain data: Shard migration |
| 255 | + nodedb-cli cluster drain-node --node-id 3 |
| 256 | + |
| 257 | + For each vShard owned by node3: |
| 258 | + - Identify target replica node (node1 or node2) |
| 259 | + - Copy data (base + WAL catch-up + atomic cut-over) |
| 260 | + - Update routing table |
| 261 | + - Data now on target node |
| 262 | + |
| 263 | +2. Remove from cluster |
| 264 | + nodedb-cli cluster remove-voter --node-id 3 |
| 265 | + |
| 266 | +3. node3 is no longer part of cluster |
| 267 | + Can be powered off |
| 268 | + |
| 269 | +4. New node can be added later (if needed) |
| 270 | + nodedb-cli cluster add-learner --node-id 4 |
| 271 | +``` |
| 272 | + |
| 273 | +## Failure Scenarios |
| 274 | + |
| 275 | +### Single Node Failure |
| 276 | + |
| 277 | +``` |
| 278 | +3-node cluster (node1 is leader): |
| 279 | + Quorum = 2 |
| 280 | + |
| 281 | +node1 crashes |
| 282 | + ├─ node1 stops sending heartbeats |
| 283 | + ├─ node2, node3 don't receive heartbeat for 300 ms |
| 284 | + ├─ node2, node3 timeout, trigger election |
| 285 | + ├─ One of (node2, node3) wins election (2 votes = quorum) |
| 286 | + ├─ New leader is elected |
| 287 | + └─ Cluster continues (reads and writes work) |
| 288 | + |
| 289 | +Recovery: node1 rejoins |
| 290 | + ├─ node1 boots, reads persistent state |
| 291 | + ├─ Discovers it was leader but term is old |
| 292 | + ├─ Applies itself as follower (new leader already exists) |
| 293 | + ├─ Catches up via replication from new leader |
| 294 | + └─ Cluster is back to 3 nodes |
| 295 | +``` |
| 296 | + |
| 297 | +### Network Partition |
| 298 | + |
| 299 | +``` |
| 300 | +3-node cluster split: |
| 301 | + Side A: node1, node2 (can talk to each other) |
| 302 | + Side B: node3 (isolated) |
| 303 | + |
| 304 | +Side A: |
| 305 | + ├─ node1, node2 have quorum (2/3) |
| 306 | + ├─ Leader stays active on side A |
| 307 | + ├─ Accepts reads and writes |
| 308 | + |
| 309 | +Side B: |
| 310 | + ├─ node3 has no quorum (1/3) |
| 311 | + ├─ node3 stops accepting writes |
| 312 | + ├─ Serves reads with EVENTUAL consistency |
| 313 | + |
| 314 | +Reconciliation: partition heals |
| 315 | + ├─ node3 reconnects |
| 316 | + ├─ node3 is behind (missed writes on side A) |
| 317 | + ├─ Followers send missing entries |
| 318 | + ├─ node3 catches up |
| 319 | + └─ Cluster is consistent again (no data loss) |
| 320 | +``` |
| 321 | + |
| 322 | +### Leader Death + Network Partition |
| 323 | + |
| 324 | +``` |
| 325 | +5-node cluster (node1 is leader): |
| 326 | + Quorum = 3 |
| 327 | + |
| 328 | +node1 dies + network partition: |
| 329 | + Side A: node2, node3 (can talk) |
| 330 | + Side B: node4, node5 (can talk) |
| 331 | + Partitioned: A <-> B |
| 332 | + |
| 333 | +Side A (quorum = 2/3, not enough): |
| 334 | + ├─ node2, node3 cannot elect leader (need 3) |
| 335 | + ├─ No leader on side A |
| 336 | + └─ Reads: allowed, Writes: blocked |
| 337 | + |
| 338 | +Side B (quorum = 2/5, not enough): |
| 339 | + ├─ node4, node5 cannot elect leader (need 3) |
| 340 | + ├─ No leader on side B |
| 341 | + └─ Reads: allowed, Writes: blocked |
| 342 | + |
| 343 | +Cluster is SPLIT-BRAIN SAFE: |
| 344 | + ├─ Neither side can write (no quorum) |
| 345 | + ├─ No conflicting writes possible |
| 346 | + ├─ When partition heals, one side is source of truth |
| 347 | +``` |
| 348 | + |
| 349 | +## Monitoring |
| 350 | + |
| 351 | +Check cluster health: |
| 352 | + |
| 353 | +```bash |
| 354 | +nodedb-cli cluster status |
| 355 | + |
| 356 | +Output: |
| 357 | +cluster_id: "abc123" |
| 358 | +node_count: 3 |
| 359 | +quorum: 2 |
| 360 | +status: HEALTHY |
| 361 | + |
| 362 | +Members: |
| 363 | + node_id: 1 |
| 364 | + role: LEADER |
| 365 | + status: ALIVE |
| 366 | + term: 100 |
| 367 | + commit_index: 50000 |
| 368 | + |
| 369 | + node_id: 2 |
| 370 | + role: FOLLOWER |
| 371 | + status: ALIVE |
| 372 | + replication_lag: 50 ms (behind leader) |
| 373 | + |
| 374 | + node_id: 3 |
| 375 | + role: FOLLOWER |
| 376 | + status: ALIVE |
| 377 | + replication_lag: 30 ms (behind leader) |
| 378 | +``` |
| 379 | + |
| 380 | +## Performance Characteristics |
| 381 | + |
| 382 | +| Operation | Single-Node | 3-Node Cluster | 5-Node Cluster | |
| 383 | +|-----------|------------|----------------|----------------| |
| 384 | +| Write latency | 1-5 ms | 5-20 ms | 5-20 ms | |
| 385 | +| Strong read | 1-5 ms | 1-5 ms | 1-5 ms | |
| 386 | +| Bounded-staleness read | < 1 ms | 1-10 ms | 1-10 ms | |
| 387 | +| Failover time | N/A | 300-500 ms | 300-500 ms | |
| 388 | +| Network overhead | 0 | ~300 Kbps | ~300 Kbps | |
| 389 | +| Max quorum loss tolerance | 0 | 1 failure | 2 failures | |
0 commit comments