Add an implementation of the Raft consensus protocol#223
Merged
Bismanpal-Singh merged 81 commits intoapache:dev-raftfrom May 3, 2026
Merged
Add an implementation of the Raft consensus protocol#223Bismanpal-Singh merged 81 commits intoapache:dev-raftfrom
Bismanpal-Singh merged 81 commits intoapache:dev-raftfrom
Conversation
fixed a race condition in deploy_local.sh
raised timeout window and heartbeat interval
Also updated ReceiveTransaction Also updated ReceiveAppendEntries
Add Raft tests
Tests now work again after adding Recovery. Addressed minor test concerns. Add Log Truncation support to Raft Recovery. This includes a new WALRecord and TruncationRecord protobuf message. Move LastLogIndex and log_ resizing into AddToLog and TruncateLog. Add PrintDebugStateLocked() so that PrintDebugState() can be called under the lock. Change Recovery to not ignore all recovery data if a portion of the data is wrong.
For checkpointing, we need to separate the logical log index and size from the actual indexing and size of the vector.
Refactor the Recovery class to pull out the code that will be used for RaftRecovery. There are now 2 derived classes, one for PBFT's Recovery and one for Raft's Recovery.
Added some handling to deal with snapshotting in Raft. Checkpoints will be taken, the last snapshot index and term need to be stored, and index arithmetic needs to be accounted for. Note that this largely is not yet in use, as the hook to actual truncate a prefix in Consensus is currently commented out. Once there is handling to send and receive Snapshots, this will be used. Added a callback hook in recovery to let the consensus protocol know when a checkpoint has finished and up to what seq it covers. Moved some Raft test header functionality into a general Raft test utility to not be attached to that specific text fixture. Added some new tests for existing Raft features. Updated lastApplied_ in Raft to be lastCommitted (meaning we have queued it to be committed). lastApplied is now tracked in Raft's Consensus class. Fixed a bug with min_seq_ tracking in Recovery. Fixed potential issues with edge cases in ReadLogsFromFiles.
This adds snapshotting to Raft, using ResilientDB's Checkpointing system. Added some handling to deal with snapshotting in Raft. Checkpoints will be taken, the last snapshot index and term need to be stored, and index arithmetic needs to be accounted for. Note that this largely is not yet in use, as the hook to actual truncate a prefix in Consensus is currently commented out. Once there is handling to send and receive Snapshots, this will be used. Added a callback hook in recovery to let the consensus protocol know when a checkpoint has finished and up to what seq it covers. Moved some Raft test header functionality into a general Raft test utility to not be attached to that specific text fixture. Added some new tests for existing Raft features. Updated lastApplied_ in Raft to be lastCommitted (meaning we have queued it to be committed). lastApplied is now tracked in Raft's Consensus class. Fixed a bug with min_seq_ tracking in Recovery. Fixed potential issues with edge cases in ReadLogsFromFiles.
Author
|
@Bismanpal-Singh I have fixed the merge conflicts. |
Contributor
|
Merged in a new development branch specific to this project. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An implementation of the Raft Consensus Protocol into ResilientDB. This implementation is still in progress. Currently missing is the ability to send and receive snapshots between replicas, as well as some additional test cases.