Skip syncing number_of_replicas when follower has auto_expand_replica…#1662
Open
monusingh-1 wants to merge 1 commit into
Open
Skip syncing number_of_replicas when follower has auto_expand_replica…#1662monusingh-1 wants to merge 1 commit into
monusingh-1 wants to merge 1 commit into
Conversation
monusingh-1
force-pushed
the
fix/1661-auto-expand-replicas-sync
branch
2 times, most recently
from
f8ed51e to
a613e4a
Compare
…s active Fixes opensearch-project#1661. When the follower index has `index.auto_expand_replicas` active (any value other than the literal "false"), the follower's local OpenSearch is responsible for deriving `index.number_of_replicas` from its own data-node count. CCR's 60s metadata sync was blindly copying the leader's `number_of_replicas` onto the follower, which in topologies where the leader has fewer data nodes than the follower caused a destructive cycle: 1. CCR pushes leader's lower `number_of_replicas` -> STARTED replica shards are destroyed on the follower. 2. OpenSearch's `adaptAutoExpandReplicas()` immediately corrects the count back up -> new UNASSIGNED shards enter peer recovery -> cluster goes YELLOW. 3. 60s later CCR syncs again -> destroys the recovering shards before recovery completes -> cycle repeats forever. Fix: in `IndexReplicationTask.pollForMetadata()`, after computing the desired settings from leader+overrides and before diffing against the follower, if the follower's `auto_expand_replicas` is active, strip `number_of_replicas` from BOTH the desired and follower settings so the diff loops neither add nor remove it. The leader's `auto_expand_replicas` itself continues to be synced, so once the user disables auto-expand on the leader the next sync cycle resumes syncing `number_of_replicas`. Any explicit `number_of_replicas` override set via the replication metadata is also suppressed while auto-expand is active on the follower; this is intentional because a fixed replica count and auto-expand are contradictory and the follower's active auto-expand takes precedence. Added unit tests for the two new companion-object helpers `isAutoExpandReplicasActive` and `filterOutNumberOfReplicas` covering: numeric range, "0-all", "false"/"False"/absent, key removal, no-op when absent, and empty settings. Signed-off-by: Monu Singh <msnghgw@amazon.com>
monusingh-1
force-pushed
the
fix/1661-auto-expand-replicas-sync
branch
from
a613e4a to
03a002a
Compare
monusingh-1
requested review from
ankitkala,
gbbafna,
krishna-ggk,
saikaranam-amazon and
soosinha
as code owners
soosinha
reviewed
Apr 28, 2026
| internal fun isAutoExpandReplicasActive(settings: Settings): Boolean { | ||
| val value = settings.get(IndexMetadata.INDEX_AUTO_EXPAND_REPLICAS_SETTING.key) ?: return false | ||
| // OpenSearch stores the disabled sentinel as the literal string "false". | ||
| return !value.equals("false", ignoreCase = true) |
Member
There was a problem hiding this comment.
Wouldnt it be simpler to check value.equals("true)
| // `number_of_replicas` are contradictory settings, and the follower's active auto-expand | ||
| // takes precedence until it is disabled. | ||
| if (isAutoExpandReplicasActive(followerSettings)) { | ||
| desiredSettings = filterOutNumberOfReplicas(desiredSettings) |
Member
There was a problem hiding this comment.
Can we add this logic above at line 543 where we are already iterating over the settings ?
There was a problem hiding this comment.
we can add it in line 535 when the desiredSettingsBuilder is being initialised. have done the same fix here in this pr #1664
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When the follower index has
index.auto_expand_replicasactive (any value other than the literal "false"), the follower's local OpenSearch is responsible for derivingindex.number_of_replicasfrom its own data-node count. CCR's 60s metadata sync was blindly copying the leader'snumber_of_replicasonto the follower, which in topologies where the leader has fewer data nodes than the follower caused a destructive cycle:number_of_replicas-> STARTED replica shards are destroyed on the follower.adaptAutoExpandReplicas()immediately corrects the count back up -> new UNASSIGNED shards enter peer recovery -> cluster goes YELLOW.Fix: in
IndexReplicationTask.pollForMetadata(), after computing the desired settings from leader+overrides and before diffing against the follower, if the follower'sauto_expand_replicasis active, stripnumber_of_replicasfrom BOTH the desired and follower settings so the diff loops neither add nor remove it. The leader'sauto_expand_replicasitself continues to be synced, so once the user disables auto-expand on the leader the next sync cycle resumes syncingnumber_of_replicas.Any explicit
number_of_replicasoverride set via the replication metadata is also suppressed while auto-expand is active on the follower; this is intentional because a fixed replica count and auto-expand are contradictory and the follower's active auto-expand takes precedence.Added unit tests for the two new companion-object helpers
isAutoExpandReplicasActiveandfilterOutNumberOfReplicascovering: numeric range, "0-all", "false"/"False"/absent, key removal, no-op when absent, and empty settings.Related Issues
Resolves #1661
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.