feat(server): Implement tagged chunk loading

[abhijat]

Changes to rdb loader to handle tagged chunks. A data stream read by the loader now looks like:

{chunk1:[stream-id]chunk-size][data...]}{untagged data...}{chunk2:[stream-id]chunk-size][data...]}

If the sender does not turn on tagging, then all the data is untagged.

where

• stream-id is 4 byte id, same for all chunks of same db+key
• chunk size is 4 byte size of data in chunk

Code changes:

• parse RDB_OPCODE_TAGGED_CHUNK and track the active chunk payload budget
• save and restore per-stream partial object state across continuation chunks
• add loader-only tests that feed handcrafted tagged byte streams, including split SBF cases and interleaving

New state in loader:

• current chunk: stream id and payload size
• stream states: partially read objects which can be resumed when next chunk for them is seen

Some of the new tests can be removed once the saver PR is merged as they will become redundant. But for now these are necessary to test the loader code in isolation by feeding it artificial byte stream containing chunks.

Example of loading one set split across multiple tagged chunks

• First chunk:
  • chunk budget is set from the tagged payload size
  • call chain: LoadKeyValPair -> ReadAndDispatchObject -> ReadObj -> ReadSet
  • reading stops when either the set is complete or the chunk budget is exhausted
• If the chunk ends first (budget exhausted):
  • the partially built PrimeValue is kept in now_chunked_
  • parser continuation state, including remaining items, is saved in stream_states_
• Continuation chunk:
  • chunk budget is set again from the new tagged payload size
  • call chain: LoadValueChunk -> ReadAndDispatchObject -> ReadObj -> ReadSet
  • parser state is restored from stream_states_
  • newly read elements are appended to the partial value in now_chunked_
• This repeats until the object is fully read, at which point:
  • continuation state is removed
  • the final value is written to the DB

[abhijat]

augment review

[augmentcode]

Review completed. 3 suggestions posted.

Comment augment review to trigger a new review at any time.

[augmentcode]

Review completed. 1 suggestion posted.

Comment augment review to trigger a new review at any time.

[augmentcode]

Review completed. 1 suggestion posted.

Comment augment review to trigger a new review at any time.

[augmentcode]

Review completed. 1 suggestion posted.

Comment augment review to trigger a new review at any time.

[augmentcode]

Review completed. 1 suggestion posted.

Comment augment review to trigger a new review at any time.

[augmentcode]

Review completed. 1 suggestion posted.

Comment augment review to trigger a new review at any time.

[augmentcode]

Review completed. 4 suggestions posted.

Comment augment review to trigger a new review at any time.

[romange]

https://mitchellh.com/writing/ghostty-leaving-github

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[romange]

@abhijat is it possible to catch in rdb_test.cc ?

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Saving StreamState copies pending_read_ (which can include large partially-filled buffers like SBF filter_data), causing an extra full-buffer copy and memory spike exactly on chunk boundaries. Prefer moving pending_read_ into StreamState (and then resetting pending_read_) to avoid duplicating potentially huge data.

Suggested change

	.pending_read = pending_read_,
	.pending_read = std::move(pending_read_),

[abhijat]

fixed

[abhijat]

@abhijat is it possible to catch in rdb_test.cc ?

yes I am trying to update the test to cover this

[abhijat]

@abhijat is it possible to catch in rdb_test.cc ?

I added in existing InterleavedLoad

• create key and hash which is on shard 1, while test runs on shard 0 (because bug was in non inlined path)
• split hash into chunks, first chunk goes in db 0
• add selectdb opcode to switch db to 1 before second chunk
• assert that hash is reassembled in db 0

with the old code it fails with

/home/abhijat/dev/cc/dragonfly/src/server/rdb_test.cc:1516: Failure
Expected equality of these values:
  Run({"HGET", key, "f1"})
    Which is: nil
  "v1"
    Which is: 0x1ab9b90

/home/abhijat/dev/cc/dragonfly/src/server/rdb_test.cc:1517: Failure
Expected equality of these values:
  Run({"HGET", key, "f2"})
    Which is: nil
  "v2"
    Which is: 0x1ab9b96

ie key, f1=v1 and f2=v2 created on db=1 which was active when last chunk was received, and not db=0, which was supposed to hold the object

with the changes it passes

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.