feat: tiered data storage#7
Open
enigbe wants to merge 7 commits into
Open
Conversation
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
9 times, most recently
from
29f47f3 to
264aa7f
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
a30cbfb to
1e7bdbc
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
5 times, most recently
from
b5e980f to
67d47c2
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
95285b0 to
4b2d345
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
2 times, most recently
from
cba29a3 to
db1fe83
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
db1fe83 to
35f9ec2
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
e89ada5 to
3abd0a5
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
5 times, most recently
from
8f122cb to
7155c40
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
4 times, most recently
from
7720935 to
181db03
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
5 times, most recently
from
5da76a8 to
dec4aa7
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
c1563e3 to
a2458e4
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
8dbb312 to
0ae61b7
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
7 times, most recently
from
cadf81f to
1f2cd4d
Compare
This commit adds `TierStore`, a tiered `KVStore` implementation that routes node persistence across three storage roles: - a primary store for durable, authoritative data - an optional backup store for a second durable copy of primary-backed data - an optional ephemeral store for rebuildable cached data such as the network graph and scorer TierStore routes ephemeral cache data to the ephemeral store when configured, while durable data remains primary+backup. Reads and lists do not consult the backup store during normal operation. For primary+backup writes and removals, this implementation treats the backup store as part of the persistence success path rather than as a best-effort background mirror. Earlier designs used asynchronous backup queueing to avoid blocking the primary path, but that weakens the durability contract by allowing primary success to be reported before backup persistence has completed. TierStore now issues primary and backup operations together and only returns success once both complete. This gives callers a clearer persistence guarantee when a backup store is configured: acknowledged primary+backup mutations have been attempted against both durable stores. The tradeoff is that dual-store operations are not atomic across stores, so an error may still be returned after one store has already been updated. Additionally, adds unit coverage for the current contract, including: - basic read/write/remove/list persistence - routing of ephemeral data away from the primary store - backup participation in the foreground success path for writes and removals
Add native builder support for tiered storage by introducing `TierStoreConfig` and builder methods for configuring ephemeral storage and a local SQLite backup mirror. During node construction, wrap the configured primary store in `TierStore` and attach secondary tiers for cache-like ephemeral data and mirrored durable backup writes. The builder constructs the backup and ephemeral stores internally using a dedicated SQLite database file. Add test coverage for full-cycle backup mirroring, same-path rejection, and UniFFI-backed builder configuration. Update `setup_builder!` so FFI-backed builder tests can use mutable configuration helpers.
…ightningdevkit#871 - Added temporary #[allow(dead_code)] annotations to NodeBuilder::set_backup_storage_dir_path and NodeBuilder::set_ephemeral_storage_dir_path - Keeps these tiered storage builder setters compilable while they remain unused before dropping in lightningdevkit#871
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
1f2cd4d to
d424891
Compare
avoid buffer clone if backup isn't configured
Here we avoid error-level logging for primary read misses and log partial primary/backup mutation failures as possible divergence.
Preserve call-time ordering for TierStore writes and removals by assigning per-key versions before returning the future. Also avoid error-level logging for primary read misses, and log partial primary/backup mutation failures as possible divergence.
Skip the Rust-only backup-store builder test when UniFFI bindings are enabled, since backup-store configuration is not exposed through FFI yet (pending lightningdevkit#871).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
We introduce
TierStore, aKVStoreimplementation that manages data acrossthree distinct storage layers.
The layers are:
(e.g., network graph). This tier aims to improve latency by leveraging a
local
KVStoredesigned for fast/local access.asynchronously/lazily to avoid blocking primary store operations.
We also permit the configuration of
Nodewith these stores allowingcallers to set exponential back-off parameters, as well as backup and ephemeral
stores, and to build the
Nodewith TierStore's primary store. These configurationoptions also extend to our foreign interface, allowing bindings target to build the
Node with their own
ffi::KVStoreimplementations.A sample Python implementation is added and tested.
Additionally, we add comprehensive testing for
TierStoreby introducingTierStorecore functionality.Nodebuilt with tiered storage.ffi::KVStoreimplementations.Concerns
It is worth considering the way retry logic is handled, especially because of nested
retries.
TierStorecomes with a basic one by default but there areKVStoreimplementationsthat come with them baked-in (e.g.
VssStore), and thus would have no need forthe wrapper-store's own logic.