store: fix restore atomicity, swapInTempDB close error, single-pass CRC32#405
Merged
bootjp merged 3 commits intofeature/fsm-kvfrom Mar 22, 2026
Merged
store: fix restore atomicity, swapInTempDB close error, single-pass CRC32#405bootjp merged 3 commits intofeature/fsm-kvfrom
bootjp merged 3 commits intofeature/fsm-kvfrom
Conversation
…pdate format comments Co-authored-by: bootjp <1306365+bootjp@users.noreply.github.com> Agent-Logs-Url: https://github.com/bootjp/elastickv/sessions/4edb95f7-2270-4eda-9b31-cd1d7fac0ddf
Copilot
AI
changed the title
[WIP] Change FSM store to Pebble for memory reduction
store: fix restore atomicity, swapInTempDB close error, single-pass CRC32
Mar 22, 2026
bootjp
approved these changes
Mar 22, 2026
| // It writes all entries into a temporary sibling directory and only swaps the | ||
| // temp directory into s.dir after a full successful read, preserving the | ||
| // existing DB on any failure. | ||
| func (s *pebbleStore) restorePebbleNativeAtomic(r io.Reader) error { |
Contributor
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
calculated cyclomatic complexity for function restorePebbleNativeAtomic is 11, max is 10 (cyclop)
| // read the spool file directly without needing a LimitReader. Entries are | ||
| // written into a temporary Pebble directory and only swapped into place after | ||
| // decoding succeeds, preserving the existing store on failure. | ||
| func (s *pebbleStore) restoreFromLegacyGob(r io.Reader) error { |
Contributor
There was a problem hiding this comment.
🚫 [golangci] reported by reviewdog 🐶
calculated cyclomatic complexity for function restoreFromLegacyGob is 13, max is 10 (cyclop)
Owner
|
@copilot fix golangci-lint |
Co-authored-by: bootjp <1306365+bootjp@users.noreply.github.com> Agent-Logs-Url: https://github.com/bootjp/elastickv/sessions/9c40de5f-6d02-45a5-a99f-9be3f8234f2c
Contributor
Author
Fixed in e1dbc8a. The |
| @@ -662,11 +663,18 @@ func (s *pebbleStore) restoreOneEntry(r io.Reader, batch *pebble.Batch) (bool, e | |||
| } | |||
|
|
|||
| func (s *pebbleStore) restoreBatchLoop(r io.Reader) error { | |||
Contributor
Author
bootjp
approved these changes
Mar 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Several correctness and durability issues in the Pebble FSM restore paths identified in review. Key problems: native Pebble restore wiped the existing DB before validating the incoming snapshot;
swapInTempDBsilently ignoreddb.Close()errors; legacy gob restore read the payload twice (spool then re-read for CRC verify); and the format history comment was misleading about pre-magic Pebble snapshots.Changes
Native Pebble restore — atomic temp-dir swap
restorePebbleNativeAtomic: writes entries into a unique temp dir (os.MkdirTemp), persistslastCommitTS, then swaps into place viaswapInTempDB— existing DB is preserved on any failure.restoreBatchLoopInto(r io.Reader, db *pebble.DB)as a standalone function so both the direct and atomic restore paths share the same batch loop without touchings.db.Restorenow dispatches native Pebble snapshots to the atomic variant instead of callingreopenFreshDBup front.swapInTempDB— check Close errorrestoreFromLegacyGob— single-pass CRC32Replaced the two-pass approach (spool-all then seek-back-verify) with a rolling
checksumSize-byte tail buffer: the CRC32 is computed incrementally during the spool, the 4-byte trailer never reaches the spool file, and the decoder reads the file once. Removed the now-deadverifyCRC32Trailerhelper. AddedspoolBufSize = 32*1024constant.Comments / backward compat clarity
restorePebbleNativedoc comment: thepebbleSnapshotMagicheader was present from day one; there are no pre-magic Pebble snapshots.Restoredefault branch clarifying that unknown streams fall to legacy gob with a clear decode error.Test coverage
Added
TestPebbleStore_Restore_NativePebbleAtomicwhich truncates a valid snapshot mid-stream and asserts the original DB contents remain accessible after the failed restore.📍 Connect Copilot coding agent with Jira, Azure Boards or Linear to delegate work to Copilot in one click without leaving your project management tool.