Skip to content

feat(tiering): Experimental list node tiering#7023

Merged
mkaruza merged 1 commit into
mainfrom
mkaruza/enable-list-node-tiered-storage
Apr 7, 2026
Merged

feat(tiering): Experimental list node tiering#7023
mkaruza merged 1 commit into
mainfrom
mkaruza/enable-list-node-tiered-storage

Conversation

@mkaruza
Copy link
Copy Markdown
Contributor

@mkaruza mkaruza commented Mar 30, 2026

Decoupline QList from TieredStorage implementation details via
TieringParams structure callbacks.

  • Introduce TieringParams struct holding function callbacks and offload state
  • Add Node::Upload() to restore an externally-loaded node back into memory
  • Add QList::SetDbIndex() to handle db reassignment, materializing offloaded
    nodes when tiering is active
  • Add StashListNode / ReadTieredListNode in TieredStorage
  • Add tiered_storage_test.cc coverage for list node offload/reload lifecycle
  • Add tiering_test.py integration tests for list tiering behavior

@mkaruza mkaruza requested review from dranikpg and romange March 30, 2026 12:43
@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 30, 2026

🤖 Augment PR Summary

Summary: This PR adds experimental tiered-storage support for QList nodes (quicklist v2) and expands test coverage around list node offload/onload/delete.

Changes:

  • Wire QList::TieringParams callbacks in list_family.cc to stash, load, and delete individual list nodes via TieredStorage.
  • Extend DB flush logic to explicitly clear QL2 lists so any offloaded nodes are cleaned up from tiered storage.
  • Add a new ListNodeTieringTest suite exercising stash/load/delete/rename/expire/flushall behaviors with tiered list nodes.
  • Add a Python integration test validating replication + RDB load correctness when tiered list nodes exist on the master.
  • Introduce/declare flags needed by the new tests (experimental list support + list tiering thresholds/compression knobs).

Technical Notes: The list-node offload path stashes node fragments independently from the top-level key, so cleanup and replication tests focus on mixed pending/offloaded states and correctness after materialization.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 2 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread src/server/list_family.cc Outdated
Comment thread src/server/list_family.cc Outdated
@mkaruza mkaruza force-pushed the mkaruza/tiered-storage-support-qlist-node branch 2 times, most recently from acad6f8 to f08e8d6 Compare March 30, 2026 12:56
@mkaruza mkaruza force-pushed the mkaruza/enable-list-node-tiered-storage branch from 654ccd6 to ce6c7c1 Compare March 30, 2026 12:57
@mkaruza mkaruza force-pushed the mkaruza/tiered-storage-support-qlist-node branch from f08e8d6 to 16b71e6 Compare March 30, 2026 13:05
@mkaruza mkaruza force-pushed the mkaruza/enable-list-node-tiered-storage branch 2 times, most recently from f458965 to b529dc4 Compare March 30, 2026 13:13
@mkaruza mkaruza force-pushed the mkaruza/tiered-storage-support-qlist-node branch 2 times, most recently from bbf2c2f to 836bd52 Compare April 1, 2026 12:50
Base automatically changed from mkaruza/tiered-storage-support-qlist-node to main April 2, 2026 19:46
@mkaruza mkaruza force-pushed the mkaruza/enable-list-node-tiered-storage branch from b529dc4 to 571a377 Compare April 2, 2026 20:00
Copilot AI review requested due to automatic review settings April 2, 2026 20:00
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds experimental tiered-storage support for QList::Node objects, wiring list node offload/onload/delete callbacks into the list implementation and extending tiered storage’s ID/addressing to support non-key-backed fragments.

Changes:

  • Extend tiered-storage identifiers and op-manager bookkeeping to support list-node fragments (ListNodeId, ReadId).
  • Implement list-node decoding/upload and integrate list tiering callbacks into list command codepaths.
  • Add C++ and Python tests covering list node offload/onload plus replication with tiered lists.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/dragonfly/tiering_test.py Adds a replication test that exercises tiered list objects.
src/server/tiering/op_manager.h Extends OwnedEntryId to include list-node identifiers.
src/server/tiering/op_manager.cc Adds ownership/stringification support for list-node IDs.
src/server/tiering/decoders.h Generalizes Decoder::Upload to void* and adds ListNodeDecoder.
src/server/tiering/decoders.cc Implements ListNodeDecoder and updates existing decoders to new Upload signature.
src/server/tiering/common.h Adds ListNodeId + ReadId variants for list-node reads.
src/server/tiered_storage.h Introduces ReadId, splits prime vs partial stashing APIs, adds list-node read/stash helpers and IsClosed().
src/server/tiered_storage.cc Implements list-node stash/upload/delete paths and adds experimental list-tiering flag.
src/server/tiered_storage_test.cc Adds ListNodeTieringTest suite covering stash/load/delete/rename/expire/flush/move scenarios.
src/server/list_family.cc Wires QList::TieringParams callbacks to tiered storage and tracks list DB index for node IDs.
src/server/hset_family.cc Updates tiered read callsites to new ReadId API.
src/server/generic_family.cc Ensures moved QList updates its DB index (materializing offloaded nodes if needed).
src/server/db_slice.cc Updates cancel-stash callsites for new API and clears QLists during DB flush to release tiered list-node segments.
src/core/tiering_types.h Adds FragmentRef::MallocUsed() API needed for stats updates.
src/core/tiering_types.cc Implements MallocUsed() for CompactValue and QList::Node fragments.
src/core/qlist.h Adds tiering callback params, DB index tracking, and node upload API for tiered restore.
src/core/qlist.cc Implements node upload, callback-based cleanup/load/offload integration, and SetDbIndex() behavior.
src/core/qlist_test.cc Removes the older minimal tiering unit test.

Comment thread src/server/tiering/decoders.cc
Comment thread src/server/list_family.cc
Comment on lines +97 to +99
if (!res) {
LOG(WARNING) << "Failed to load list node from tiered storage: " << res.error().message();
}
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If ReadTieredListNode() fails, the node likely remains offloaded (node->entry still not materialized) but the list operation will continue after only logging a warning; this can violate QList::Materialize() expectations and lead to crashes/corruption. Consider making load failures fatal (DFATAL) or ensuring the node is put into a safe in-memory state before returning (and/or propagate an error back to the command).

Suggested change
if (!res) {
LOG(WARNING) << "Failed to load list node from tiered storage: " << res.error().message();
}
CHECK(res) << "Failed to load list node from tiered storage: " << res.error().message();

Copilot uses AI. Check for mistakes.
Comment thread src/server/tiered_storage.cc
@mkaruza mkaruza force-pushed the mkaruza/enable-list-node-tiered-storage branch from 571a377 to 7d32a02 Compare April 2, 2026 21:22
@mkaruza mkaruza changed the title feat(tiering): Experimental QList::Node tiered storage feat(tiering): Experimental list node tiering Apr 2, 2026
dranikpg
dranikpg previously approved these changes Apr 3, 2026
Comment thread tests/dragonfly/tiering_test.py
Comment thread src/server/tiered_storage.h
Decoupline QList from TieredStorage implementation details via
TieringParams structure callbacks.

- Introduce TieringParams struct holding function callbacks and offload state
- Add Node::Upload() to restore an externally-loaded node back into memory
- Add QList::SetDbIndex() to handle db reassignment, materializing offloaded
  nodes when tiering is active
- Add StashListNode / ReadTieredListNode in TieredStorage
- Add tiered_storage_test.cc coverage for list node offload/reload lifecycle
- Add tiering_test.py integration tests for list tiering behavior

Signed-off-by: mkaruza <mario@dragonflydb.io>
@mkaruza mkaruza force-pushed the mkaruza/enable-list-node-tiered-storage branch from 7d32a02 to efdc68e Compare April 7, 2026 06:34
@mkaruza mkaruza merged commit 4f61ab4 into main Apr 7, 2026
12 checks passed
@mkaruza mkaruza deleted the mkaruza/enable-list-node-tiered-storage branch April 7, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants