feat(tiering): Add experimental list node offloading by mkaruza · Pull Request #6944 · dragonflydb/dragonfly

mkaruza · 2026-03-22T14:55:32Z

Extend tiered storage to offload individual QList nodes. Offloading is supported
for RAW node objects. Functionality is controlled with flag
--tiered_experimental_list_support.

Key changes:

Add ListNodeId (dbid, ql*, node*) as a new tiering identity type
alongside KeyRef; introduce PendingId/ReadId variants so OpManager
can handle both key-level and node-level I/O uniformly.
Add tiering callbacks (offload_cb, onload_cb, delete_cb) to
QList::TieringParams. PromoteToQLIfNeeded now wires them up when
tiered storage is available, using the new db_id_ member on
ListWrapper to store DbIndex for usage.
Update CancelStash / ReadInternal signatures to accept the generic
PendingId / ReadId types instead of (dbid, key).
Expose StashListNode / ReadTieredListNode free functions used by the
ListWrapper callbacks.
Add QList::Materialize that is used during RDB save to fetch back data in memory
from tiered storage.
Added ListNodeTieringTest suite. Replication test for tiered list objects.

augmentcode · 2026-03-22T14:59:41Z

🤖 Augment PR Summary

Summary: Adds experimental tiered-storage support for offloading individual QList nodes (list internals) to disk, gated by --tiered_experimental_list_support.

Changes:

Extends compact/external representations with ExternalRep::LIST_NODE and maps it to OBJ_LIST.
Introduces list-node tiering identities (ListNodeId) and expands PendingId/ReadId so OpManager can track key-level and node-level I/O uniformly.
Adds per-node tiering hooks to QList (offload_cb/onload_cb/delete_cb) and updates QList internals to handle offloaded/io_pending nodes.
Implements partial-value stashing/reading in TieredStorage for list nodes, with separate accounting and lifecycle handling.
Wires list tiering callbacks from list_family when tiered storage is available and list tiering threshold is enabled.
Adds QList::Materialize and uses it during RDB save to ensure offloaded nodes are brought back into memory for serialization.
Adds extensive tiered list-node test coverage (including compressed-node scenarios) in tiered_storage_test.cc.

Technical Notes: List-node offloading is currently constrained (e.g. page-occupancy checks) and uses pointer-based identities (QList*/Node*) for tracking async I/O and cleanup.

_{🤖 Was this summary useful? React with 👍 or 👎}

augmentcode

Review completed. 4 suggestions posted.

Comment augment review to trigger a new review at any time.

augmentcode

Review completed. 2 suggestions posted.

Comment augment review to trigger a new review at any time.

Copilot

Pull request overview

Extends tiered storage beyond key-level PrimeValues to support experimental offloading of individual QList::Node payloads to disk, gated by --tiered_experimental_list_support, by introducing a node-identity (ListNodeId) and wiring list-tiering callbacks into QList.

Changes:

Introduces ListNodeId plus PendingId/ReadId variants so tiering I/O can address both keys and list nodes uniformly.
Adds list-node offload/onload/delete callback hooks (QList::TieringParams) and connects them from list_family when tiered storage is available.
Adds a new test suite (ListNodeTieringTest) to validate list node stash/load/delete behaviors.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
src/server/tiering/common.h	Adds `ListNodeId`, expands `PendingId`, introduces `ReadId`.
src/server/tiering/op_manager.h	Extends `OwnedEntryId` to include `ListNodeId`.
src/server/tiering/op_manager.cc	Supports owning/logging of `ListNodeId`.
src/server/tiered_storage.h	Generalizes `Read`/`CancelStash`, adds list-node helpers, adds `IsClosed()`.
src/server/tiered_storage.cc	Implements list-node stash/read paths and feature gating flag wiring.
src/server/list_family.cc	Wires tiering callbacks into `QList` via `ListWrapper`/promotion.
src/core/qlist.h	Adds node `io_pending`, external record, and tiering callbacks.
src/core/qlist.cc	Implements offload/onload/delete lifecycle hooks and `Node::SetExternal`.
src/core/tiering_types.h / .cc	Extends `FragmentRef` to support `QList::Node*` fragments.
src/core/compact_object.h / .cc	Adds `ExternalRep::LIST_NODE` and object-type mapping behavior.
src/server/tiered_storage_test.cc	Adds list node tiering test coverage.
src/server/hset_family.cc	Adapts tiered `Read` call to new signature.
src/server/db_slice.cc	Adapts `CancelStash` calls to new `PendingId` signature.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 9 comments.

mkaruza · 2026-03-23T17:54:52Z

augment review

augmentcode

Review completed. 4 suggestions posted.

Comment augment review to trigger a new review at any time.

dranikpg · 2026-03-24T10:31:06Z

wow this is a lot of code 😟

dranikpg

It's really hard to review such a large PR as if its "ready to be commited" code. I know it's very cumbersome to split all of this. I usually make different commits and there are lots of refactoring parts here (uniting db_index, key), extending ListWrapper, etc.

I will take a look a little later at the other parts. I'm not much in context of list tiering yet

romange · 2026-03-24T12:37:34Z

Please rebase and resolve conflicts.

romange · 2026-03-24T12:37:52Z

augment review

mkaruza · 2026-03-25T21:17:25Z

augment review

augmentcode

Review completed. 3 suggestions posted.

Comment augment review to trigger a new review at any time.

Extend tiered storage to offload individual QList nodes to disk, gated by --tiered_experimental_list_support. Key changes: - Add ListNodeId (dbid, node*, ql*) as a new tiering identity type alongside KeyRef; introduce PendingId/ReadId variants so OpManager can handle both key-level and node-level I/O uniformly. - Add tiering callbacks (offload_cb, onload_cb, delete_cb) to QList::TieringParams. PromoteToQLIfNeeded now wires them up when tiered storage is available, using the new db_id_ member on ListWrapper to store DbIndex for usage. - Update CancelStash / ReadInternal signatures to accept the generic PendingId / ReadId types instead of (dbid, key). - Add IsClosed() / is_closed_ guard so delete_cb skips I/O after TieredStorage::Close() is called. - Expose StashListNode / ReadTieredListNode free functions used by the ListWrapper callbacks. - Added ListNodeTieringTest suite Signed-off-by: mkaruza <mario@dragonflydb.io>

Don't stash list nodes if don't occupty whole page (either raw or compressed).

romange · 2026-03-30T11:03:51Z

It's really hard to review such a large PR as if its "ready to be commited" code. I know it's very cumbersome to split all of this. I usually make different commits and there are lots of refactoring parts here (uniting db_index, key), extending ListWrapper, etc.

I will take a look a little later at the other parts. I'm not much in context of list tiering yet

I join the sentiment. There is no reason to review it as a whole. I should be split into 3 at least. Lets start with src/core/ changes, first, then tiering and only then - families.

mkaruza · 2026-03-30T12:59:03Z

@romange @dranikpg

I have split this PR into 3:

#7021 extend QList::Node to support tiering.
#7022 extend TeieredStorage to handle QList::Node
#7023 Enables tiering of nodes and adds test

PR's needs to be merged in order. I'll close this once they are review/merged

mkaruza · 2026-04-07T07:57:55Z

Closing: #7021 #7022 #7023 were merged

mkaruza marked this pull request as draft March 22, 2026 14:55

augmentcode Bot reviewed Mar 22, 2026

View reviewed changes

Comment thread src/core/qlist.cc Outdated

Comment thread src/core/qlist.cc

Comment thread src/server/list_family.cc Outdated

Comment thread src/server/list_family.cc Outdated

mkaruza force-pushed the mkaruza/tiering-fragment branch from 4f08cf6 to e708544 Compare March 22, 2026 15:00

mkaruza force-pushed the mkaruza/github#6467 branch 2 times, most recently from 46df68e to 3517947 Compare March 23, 2026 13:02

mkaruza changed the title ~~feat(tiering): Support for partial offloading of LIST nodes~~ feat(tiering): Add experimental list node offloading Mar 23, 2026

mkaruza marked this pull request as ready for review March 23, 2026 13:04

mkaruza force-pushed the mkaruza/github#6467 branch from 3517947 to 452b5f7 Compare March 23, 2026 13:04

mkaruza changed the base branch from mkaruza/tiering-fragment to main March 23, 2026 13:05

Copilot AI review requested due to automatic review settings March 23, 2026 13:08

mkaruza force-pushed the mkaruza/github#6467 branch from 452b5f7 to 27d668b Compare March 23, 2026 13:08

Copilot started reviewing on behalf of mkaruza March 23, 2026 13:09 View session

augmentcode Bot reviewed Mar 23, 2026

View reviewed changes

Comment thread src/server/list_family.cc Outdated

Comment thread src/core/tiering_types.cc Outdated

Copilot AI reviewed Mar 23, 2026

View reviewed changes

mkaruza force-pushed the mkaruza/github#6467 branch from 77572ce to 7e7edfe Compare March 23, 2026 15:39

mkaruza requested a review from Copilot March 23, 2026 17:09

Copilot started reviewing on behalf of mkaruza March 23, 2026 17:09 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

augmentcode Bot reviewed Mar 23, 2026

View reviewed changes

Comment thread src/core/compact_object.cc Outdated

Comment thread src/core/qlist.h Outdated

Comment thread src/core/qlist.cc Outdated

Comment thread src/server/list_family.cc Outdated

mkaruza force-pushed the mkaruza/github#6467 branch 3 times, most recently from 61cdbc9 to 90afdd0 Compare March 24, 2026 06:30

mkaruza requested review from dranikpg and romange March 24, 2026 07:50

dranikpg reviewed Mar 24, 2026

View reviewed changes

Comment thread src/server/tiered_storage.h

Comment thread src/server/tiered_storage.cc Outdated

Comment thread src/server/tiered_storage.cc Outdated

Comment thread src/server/tiering/op_manager.h Outdated

Comment thread src/server/list_family.cc Outdated

mkaruza force-pushed the mkaruza/github#6467 branch 3 times, most recently from ad4aa44 to 622c609 Compare March 25, 2026 16:26

augmentcode Bot reviewed Mar 25, 2026

View reviewed changes

Comment thread src/core/qlist.cc Outdated

Comment thread src/core/qlist.cc

Comment thread src/core/qlist.cc

mkaruza force-pushed the mkaruza/github#6467 branch from 36f213f to 27f7adc Compare March 25, 2026 21:39

mkaruza requested review from dranikpg and romange March 26, 2026 08:42

mkaruza force-pushed the mkaruza/github#6467 branch 2 times, most recently from 5e50251 to fb8f92a Compare March 27, 2026 08:31

dranikpg previously approved these changes Mar 27, 2026

View reviewed changes

Comment thread src/core/tiering_types.cc Outdated

Comment thread src/server/list_family.cc Outdated

mkaruza dismissed dranikpg’s stale review via 09b209a March 27, 2026 15:02

mkaruza requested a review from dranikpg March 27, 2026 15:03

dranikpg previously approved these changes Mar 27, 2026

View reviewed changes

mkaruza dismissed dranikpg’s stale review via d4572a5 March 30, 2026 07:29

mkaruza force-pushed the mkaruza/github#6467 branch from 09b209a to d4572a5 Compare March 30, 2026 07:29

mkaruza requested a review from dranikpg March 30, 2026 07:32

romange reviewed Mar 30, 2026

View reviewed changes

Comment thread src/core/qlist.h Outdated

mkaruza added 6 commits March 30, 2026 10:52

Review Changes

dbe206f

PR review changes

d318a8a

Don't stash list nodes if don't occupty whole page (either raw or compressed).

Changes after rebase on main with ZSTD functionality

3e39352

Revert offlading around compressed nodes. Added replication list test

5d4f5db

Use anonymous union

7316313

mkaruza force-pushed the mkaruza/github#6467 branch from d4572a5 to 7316313 Compare March 30, 2026 09:39

mkaruza closed this Apr 7, 2026

mkaruza deleted the mkaruza/github#6467 branch April 7, 2026 07:59

Conversation

mkaruza commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

augmentcode Bot commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkaruza commented Mar 23, 2026

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dranikpg commented Mar 24, 2026

Uh oh!

dranikpg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

romange commented Mar 24, 2026

Uh oh!

romange commented Mar 24, 2026

Uh oh!

mkaruza commented Mar 25, 2026

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkaruza commented Mar 22, 2026 •

edited

Loading

augmentcode Bot commented Mar 22, 2026 •

edited

Loading

mkaruza commented Mar 30, 2026 •

edited

Loading