feat(cluster): expose slot migration progress in CLUSTER INFO#3543
Closed
git-hulk wants to merge 1 commit into
Closed
feat(cluster): expose slot migration progress in CLUSTER INFO#3543git-hulk wants to merge 1 commit into
git-hulk wants to merge 1 commit into
Conversation
Expose the detailed progress of the ongoing slot migration task via the CLUSTER INFO command while the migration state is 'start': - migrating_stage: start|snapshot|wal|success|failed|clean - migrating_elapsed_ms - migrating_keys_migrated/expired/empty (redis-command type) - migrating_sent_bytes/batches/entries - migrating_rate_kbps - migrating_wal_sequence_gap Also make current_stage_ and wal_begin_seq_ atomic since they are now read from other threads while the migration thread updates them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
I haven't reviewed this PR yet, will reopen after that. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What does this PR do?
Currently, while a slot migration is running,
CLUSTER INFOonly reportsmigrating_slot(s),destination_node, andmigrating_state: start. All progress details (key counters, sent bytes/batches, WAL catch-up) exist internally but are only written to server logs at the end of each phase, so operators have no way to observe an in-flight migration or estimate how far along it is.This PR exposes the detailed progress of the ongoing migration task via
CLUSTER INFOwhile the migration state isstart:migrating_stage: the current stage of the state machine (start/snapshot/wal/success/failed/clean).migrating_keys_*: snapshot key counters, only counted by the redis-command migration type.migrating_sent_bytes/batches/entries: data sent to the destination node. For the redis-command type, a batch is a pipeline of commands and an entry is a command; for the raw-key-value type, a batch is an APPLYBATCH write batch and an entry is a write batch operation.migrating_wal_sequence_gap: the distance between the latest sequence number and the WAL sequence the migration has caught up to, i.e. the pending incremental data.The existing fields are unchanged, and the new fields are only emitted while the task is ongoing, so existing tools polling
migrating_stateare unaffected. All counters are emitted in both migration types (zeros where a type doesn't use them) so that parsers don't need to be aware of themigrate-typeconfig.This also makes
current_stage_andwal_begin_seq_atomic since they are now read from other threads while the migration thread updates them (current_stage_was already read cross-thread byServer::WaitNoMigrateProcessing).Testing
Added two integration tests that hold a migration in flight (via
migrate-speed/migrate-batch-rate-limit-mb) and verify the progress counters go non-zero during the migration for both migration types, and that the fields disappear after the task finishes.🤖 Generated with Claude Code