feat(tiering): Experimental list node tiering#7023
Conversation
🤖 Augment PR SummarySummary: This PR adds experimental tiered-storage support for Changes:
Technical Notes: The list-node offload path stashes node fragments independently from the top-level key, so cleanup and replication tests focus on mixed pending/offloaded states and correctness after materialization. 🤖 Was this summary useful? React with 👍 or 👎 |
acad6f8 to
f08e8d6
Compare
654ccd6 to
ce6c7c1
Compare
f08e8d6 to
16b71e6
Compare
f458965 to
b529dc4
Compare
bbf2c2f to
836bd52
Compare
b529dc4 to
571a377
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds experimental tiered-storage support for QList::Node objects, wiring list node offload/onload/delete callbacks into the list implementation and extending tiered storage’s ID/addressing to support non-key-backed fragments.
Changes:
- Extend tiered-storage identifiers and op-manager bookkeeping to support list-node fragments (
ListNodeId,ReadId). - Implement list-node decoding/upload and integrate list tiering callbacks into list command codepaths.
- Add C++ and Python tests covering list node offload/onload plus replication with tiered lists.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/dragonfly/tiering_test.py | Adds a replication test that exercises tiered list objects. |
| src/server/tiering/op_manager.h | Extends OwnedEntryId to include list-node identifiers. |
| src/server/tiering/op_manager.cc | Adds ownership/stringification support for list-node IDs. |
| src/server/tiering/decoders.h | Generalizes Decoder::Upload to void* and adds ListNodeDecoder. |
| src/server/tiering/decoders.cc | Implements ListNodeDecoder and updates existing decoders to new Upload signature. |
| src/server/tiering/common.h | Adds ListNodeId + ReadId variants for list-node reads. |
| src/server/tiered_storage.h | Introduces ReadId, splits prime vs partial stashing APIs, adds list-node read/stash helpers and IsClosed(). |
| src/server/tiered_storage.cc | Implements list-node stash/upload/delete paths and adds experimental list-tiering flag. |
| src/server/tiered_storage_test.cc | Adds ListNodeTieringTest suite covering stash/load/delete/rename/expire/flush/move scenarios. |
| src/server/list_family.cc | Wires QList::TieringParams callbacks to tiered storage and tracks list DB index for node IDs. |
| src/server/hset_family.cc | Updates tiered read callsites to new ReadId API. |
| src/server/generic_family.cc | Ensures moved QList updates its DB index (materializing offloaded nodes if needed). |
| src/server/db_slice.cc | Updates cancel-stash callsites for new API and clears QLists during DB flush to release tiered list-node segments. |
| src/core/tiering_types.h | Adds FragmentRef::MallocUsed() API needed for stats updates. |
| src/core/tiering_types.cc | Implements MallocUsed() for CompactValue and QList::Node fragments. |
| src/core/qlist.h | Adds tiering callback params, DB index tracking, and node upload API for tiered restore. |
| src/core/qlist.cc | Implements node upload, callback-based cleanup/load/offload integration, and SetDbIndex() behavior. |
| src/core/qlist_test.cc | Removes the older minimal tiering unit test. |
| if (!res) { | ||
| LOG(WARNING) << "Failed to load list node from tiered storage: " << res.error().message(); | ||
| } |
There was a problem hiding this comment.
If ReadTieredListNode() fails, the node likely remains offloaded (node->entry still not materialized) but the list operation will continue after only logging a warning; this can violate QList::Materialize() expectations and lead to crashes/corruption. Consider making load failures fatal (DFATAL) or ensuring the node is put into a safe in-memory state before returning (and/or propagate an error back to the command).
| if (!res) { | |
| LOG(WARNING) << "Failed to load list node from tiered storage: " << res.error().message(); | |
| } | |
| CHECK(res) << "Failed to load list node from tiered storage: " << res.error().message(); |
571a377 to
7d32a02
Compare
Decoupline QList from TieredStorage implementation details via TieringParams structure callbacks. - Introduce TieringParams struct holding function callbacks and offload state - Add Node::Upload() to restore an externally-loaded node back into memory - Add QList::SetDbIndex() to handle db reassignment, materializing offloaded nodes when tiering is active - Add StashListNode / ReadTieredListNode in TieredStorage - Add tiered_storage_test.cc coverage for list node offload/reload lifecycle - Add tiering_test.py integration tests for list tiering behavior Signed-off-by: mkaruza <mario@dragonflydb.io>
7d32a02 to
efdc68e
Compare
Decoupline QList from TieredStorage implementation details via
TieringParams structure callbacks.
nodes when tiering is active