scan_reader: parallel dump + batched sends + bounded queue; redis_writer: parallel connections#1056
Draft
oOraph wants to merge 2 commits into
Draft
scan_reader: parallel dump + batched sends + bounded queue; redis_writer: parallel connections#1056oOraph wants to merge 2 commits into
oOraph wants to merge 2 commits into
Conversation
Throughput and memory improvements for the scan_reader (SCAN+DUMP+PTTL+RESTORE) path and the standalone redis_writer. On a ~38M-key standalone source, end-to-end scan-based sync goes from ~90k to ~280k keys/s with flat memory. All new options default to the prior behavior except the scan queue bound (see below), which fixes a latent OOM. - redis_writer: add 'parallel' (default 1) — fan RESTOREs across N target connections draining a shared channel. A single connection was the cap while the target had ample headroom. - scan_reader: add 'dump_parallel' (default 1) — N source connections each run DUMP+PTTL, draining the shared scan queue (the Go channel hands each key to exactly one worker; no key partitioning needed). - scan_reader: batch DUMP/PTTL sends — add client.SendNoFlush(); dump() buffers source commands and flushes every 128 keys or 5ms instead of Send()'s flush-per-command. CPU profiling showed ~64% of time in socket syscalls; this was the single biggest win. - scan_reader: bound the scan->dump queue via 'dump_queue_size' (default 100000, previously hardcoded to 100000000). The old value buffered ~the whole keyspace in RAM (OOM on large DBs) and the large live heap also degraded throughput via GC. Set a large value to restore the old behavior.
|
|
…commands The loader unconditionally collected every Rewrite() command into a slice before deciding RESTORE-vs-individual, so a single huge collection (a 120M-member zset in our data) materialized the whole raw value plus all rewrite commands in memory and OOMed the loader (~54GiB). Cap the raw-value capture at target_redis_proto_max_bulk_len via a small io.Writer: while a value fits the cap, buffer its commands (it may RESTORE); once it overflows, flush the buffer and stream the remaining commands directly, dropping the raw bytes. Peak heap is now bounded by the threshold regardless of object size. Small values still replay as a single RESTORE; oversized ones stream as individual commands (Rewrite's leading del gives replace semantics).
oOraph
force-pushed
the
bound_scan_reader_dump_queue
branch
from
1fb5dc1 to
9e7d7be
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Throughput & memory improvements for the
scan_reader(SCAN + DUMP + PTTL + RESTORE) path and the standaloneredis_writer. On a ~40M-key standalone source, end-to-end scan-based sync went from ~25k to ~280k keys/s with flat memory.Motivation
Migrating a large keyspace (tens of millions of keys) via
scan_reader— e.g. from a server that can't servePSYNC, like Dragonfly — was throughput- and memory-limited:DUMP+PTTLper key caps ~22k keys/s on round-trip latency.client.Send()flushes after every command, so each key costs 2 source send syscalls; CPU profiling showed ~64% of time in socket syscalls.needDumpQueueis hardcoded to100_000_000, so on a large DB it buffers ~the whole keyspace in RAM (OOM), and the large live heap also degrades throughput via GC during scan-fill.Changes
redis_writer.parallel(default1, unchanged behavior) — fan RESTOREs across N target connections draining a shared channel.scan_reader.dump_parallel(default1, unchanged behavior) — N source connections running DUMP+PTTL, draining the shared scan queue (the Go channel hands each key to exactly one worker — no key partitioning).DUMP/PTTLsends — addclient.SendNoFlush();dump()buffers source commands and flushes every 128 keys or 5ms instead of per-command. Amortizes hundreds of commands per syscall — the single biggest win. Internal only (no API change); preserves command/reply ordering.scan_reader.dump_queue_size(default100000, was an effectively-unbounded hardcoded100000000) — bound the scan→dump handoff queue. The old value buffered ~the whole keyspace in RAM (OOM risk on large DBs) and the oversized live heap hurt throughput via GC; the bounded default backpressuresscan()todump()speed and keeps memory flat. Set a large value to restore the previous behavior.Benchmark (Tens of millions of keys, standalone → standalone)
count=50000Memory stays ~50 MiB with the bounded queue (vs many GiB unbounded).
Compatibility
parallelanddump_paralleldefault to1(no change). The one default change isdump_queue_size(100000 vs the old hardcoded 100000000): it caps the in-flight scan buffer — throughput is unaffected or improved (benchmarks above), only peak memory changes (for the better). Setdump_queue_sizelarge to opt back into the old unbounded behavior.