kv/kvserver: TestRebalancingAndCrossRegionZoneSnapshotMetrics failed

[cockroach-teamcity]

kv/kvserver.TestRebalancingAndCrossRegionZoneSnapshotMetrics failed on release-26.1 @ ec883a1305620cb561809f29464430ecfd2ade2c:

        	            	expected: map[string]kvserver_test.snapshotBytesMetrics{"":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:599}, ".cross-region":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:599}, ".cross-zone":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:0}, ".rebalancing":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:599}, ".recovery":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:0}}
        	            	actual  : map[string]kvserver_test.snapshotBytesMetrics{"":kvserver_test.snapshotBytesMetrics{sentBytes:822, rcvdBytes:599}, ".cross-region":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:599}, ".cross-zone":kvserver_test.snapshotBytesMetrics{sentBytes:822, rcvdBytes:0}, ".rebalancing":kvserver_test.snapshotBytesMetrics{sentBytes:822, rcvdBytes:599}, ".recovery":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:0}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,3 +2,3 @@
        	            	  (string) "": (kvserver_test.snapshotBytesMetrics) {
        	            	-  sentBytes: (int64) 0,
        	            	+  sentBytes: (int64) 822,
        	            	   rcvdBytes: (int64) 599
        	            	@@ -10,3 +10,3 @@
        	            	  (string) (len=11) ".cross-zone": (kvserver_test.snapshotBytesMetrics) {
        	            	-  sentBytes: (int64) 0,
        	            	+  sentBytes: (int64) 822,
        	            	   rcvdBytes: (int64) 0
        	            	@@ -14,3 +14,3 @@
        	            	  (string) (len=12) ".rebalancing": (kvserver_test.snapshotBytesMetrics) {
        	            	-  sentBytes: (int64) 0,
        	            	+  sentBytes: (int64) 822,
        	            	   rcvdBytes: (int64) 599
        	Test:       	TestRebalancingAndCrossRegionZoneSnapshotMetrics/first_receiver
--- FAIL: TestRebalancingAndCrossRegionZoneSnapshotMetrics/first_receiver (0.00s)
=== RUN   TestRebalancingAndCrossRegionZoneSnapshotMetrics/sender
    replica_learner_test.go:2420: 
        	Error Trace:	pkg/kv/kvserver/replica_learner_test.go:2420
        	Error:      	Not equal: 
        	            	expected: map[string]kvserver_test.snapshotBytesMetrics{"":kvserver_test.snapshotBytesMetrics{sentBytes:1421, rcvdBytes:0}, ".cross-region":kvserver_test.snapshotBytesMetrics{sentBytes:599, rcvdBytes:0}, ".cross-zone":kvserver_test.snapshotBytesMetrics{sentBytes:822, rcvdBytes:0}, ".rebalancing":kvserver_test.snapshotBytesMetrics{sentBytes:1421, rcvdBytes:0}, ".recovery":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:0}}
        	            	actual  : map[string]kvserver_test.snapshotBytesMetrics{"":kvserver_test.snapshotBytesMetrics{sentBytes:599, rcvdBytes:0}, ".cross-region":kvserver_test.snapshotBytesMetrics{sentBytes:599, rcvdBytes:0}, ".cross-zone":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:0}, ".rebalancing":kvserver_test.snapshotBytesMetrics{sentBytes:599, rcvdBytes:0}, ".recovery":kvserver_test.snapshotBytesMetrics{sentBytes:0, rcvdBytes:0}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,3 +2,3 @@
        	            	  (string) "": (kvserver_test.snapshotBytesMetrics) {
        	            	-  sentBytes: (int64) 1421,
        	            	+  sentBytes: (int64) 599,
        	            	   rcvdBytes: (int64) 0
        	            	@@ -10,3 +10,3 @@
        	            	  (string) (len=11) ".cross-zone": (kvserver_test.snapshotBytesMetrics) {
        	            	-  sentBytes: (int64) 822,
        	            	+  sentBytes: (int64) 0,
        	            	   rcvdBytes: (int64) 0
        	            	@@ -14,3 +14,3 @@
        	            	  (string) (len=12) ".rebalancing": (kvserver_test.snapshotBytesMetrics) {
        	            	-  sentBytes: (int64) 1421,
        	            	+  sentBytes: (int64) 599,
        	            	   rcvdBytes: (int64) 0
        	Test:       	TestRebalancingAndCrossRegionZoneSnapshotMetrics/sender
--- FAIL: TestRebalancingAndCrossRegionZoneSnapshotMetrics/sender (0.00s)

Parameters:

• attempt=1
• run=1
• shard=18

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-62206

[arulajmani]

/investigate

[github-actions]

Investigation: TestRebalancingAndCrossRegionZoneSnapshotMetrics

Investigated failure: issue body
Failure SHA: ec883a1305620cb561809f29464430ecfd2ade2c
Confidence: moderate

What This Test Does

The test verifies that snapshot byte metrics (range.snapshots.{rebalancing,cross-region,cross-zone}.{sent,rcvd}-bytes) are correctly tracked when sending learner snapshots across a 3-node cluster with different localities. It sends two snapshots: a cross-region snapshot from server[0] (us-east/us-east-1) to server[1] (us-west/us-west-1), and a cross-zone snapshot from server[0] to server[2] (us-east/us-east-2). It then asserts that the metrics on each server match the expected values.

Where the Failure Occurs

The test fails in two subtests at replica_learner_test.go:2420 (sender) and replica_learner_test.go:2437 (first_receiver):

• first_receiver (server[1]): sentBytes is 822 instead of expected 0. Server[1] unexpectedly sent a snapshot.
• sender (server[0]): sentBytes is 599 instead of expected 1421. Server[0] only sent one snapshot instead of two.

The values 599 + 822 = 1421 = totalSnapshotLength, confirming that the second snapshot (822 bytes) was sent by server[1] instead of server[0].

Analysis

Hypothesis: Snapshot delegation caused the second snapshot to be sent by server[1] instead of server[0]

The test does not disable the kv.snapshot_delegation.max_delegation_attempts cluster setting, which defaults to 1 (see replica_command.go:3192-3202). This means snapshot delegation is enabled.

When the second snapshot is sent (adding server[2] as a voter), sendSnapshotUsingDelegate (replica_command.go:2945) calls getSenderReplicas (replica_command.go:2723) to determine who should send the snapshot. At this point, server[1] is a non-voter on the range and is a candidate delegate.

The getSenderReplicas function filters candidates by storePool.IsStoreHealthy() (replica_command.go:2752-2756). Normally, the coordinator (server[0]) would be in the candidate list and would be selected because it has the lowest diversity score to the recipient (0.5 vs 1.0 for server[1]). However, if server[0]'s store is momentarily considered unhealthy by its own store pool (e.g., due to a liveness heartbeat delay causing StoreStatusSuspect or StoreStatusUnknown — see store_pool.go:210-221), it would be excluded from the candidates list.

With server[0] excluded, server[1] becomes the only candidate and is selected as the delegate. The coordinator (server[0]) is still appended as a fallback at the end of the sender list (replica_command.go:2819), but since server[1] is tried first and succeeds, the snapshot is sent from server[1].

When server[1] sends the snapshot via followerSendSnapshot, the sent-bytes metrics are recorded on server[1]'s store (replica_command.go:3359-3387), and the cross-region/cross-zone classification uses the coordinator-to-recipient locality comparison (server[0] to server[2] = cross-zone), which explains why the 822 bytes appear under .cross-zone on server[1].

Supporting evidence:

1. The failure pattern exactly matches what delegation would produce: server[1] has 822 sent bytes (= secSnapshotLength), and server[0] is missing those same bytes.
2. The .cross-zone classification on server[1]'s sent bytes is consistent with the coordinator (server[0], us-east) to recipient (server[2], us-east) comparison, NOT the actual sender (server[1], us-west) to recipient. This matches the comparisonResult logic at replica_command.go:3356-3357 which uses req.CoordinatorReplica.NodeID.
3. The test does NOT set NumDelegateLimit to 0, unlike other tests (e.g., line 267 of the same file).

Recommended fix:
Disable snapshot delegation in this test by adding the following after the test cluster is started:

kvserver.NumDelegateLimit.Override(ctx, &tc.Server(0).ClusterSettings().SV, 0)

This ensures the coordinator always sends snapshots directly, eliminating the delegation flake. This approach is already used in other tests in the same file (line 267).

Confidence: moderate — The delegation hypothesis fully explains the failure pattern and metric values, but without EngFlow test logs (inaccessible during this investigation), we cannot confirm the exact store pool status that triggered the delegation.

Related Issues and PRs

• kv/kvserver: TestRebalancingAndCrossRegionZoneSnapshotMetrics failed #137249 (closed): Same test, different failure mode (byte count mismatch due to lease upgrade race, fixed by f12c530076b). That fix addressed a different race condition (lease type affecting snapshot size). The current failure has a fundamentally different signature (wrong server sending the snapshot).

Timeline

• The delegation feature (getSenderReplicas / NumDelegateLimit) has been in the codebase for a while and defaults to 1.
• The test was last modified by f12c530076b (Dec 2024) to fix a lease upgrade race, but that fix did not address the delegation issue.
• The test has always been susceptible to this delegation flake, but it requires specific timing conditions (store pool health check race) to trigger.

Detailed evidence and investigation notes

Log Excerpts

From the issue body — first_receiver subtest failure:

expected: map[string]kvserver_test.snapshotBytesMetrics{
  "":             {sentBytes:0, rcvdBytes:599},
  ".cross-region":{sentBytes:0, rcvdBytes:599},
  ".cross-zone":  {sentBytes:0, rcvdBytes:0},
  ".rebalancing": {sentBytes:0, rcvdBytes:599},
  ".recovery":    {sentBytes:0, rcvdBytes:0}
}
actual: map[string]kvserver_test.snapshotBytesMetrics{
  "":             {sentBytes:822, rcvdBytes:599},
  ".cross-region":{sentBytes:0, rcvdBytes:599},
  ".cross-zone":  {sentBytes:822, rcvdBytes:0},
  ".rebalancing": {sentBytes:822, rcvdBytes:599},
  ".recovery":    {sentBytes:0, rcvdBytes:0}
}

Key observations:

• sentBytes: 822 appears on the "", .cross-zone, and .rebalancing metrics — exactly the metrics that would be incremented for a cross-zone rebalancing snapshot.
• rcvdBytes: 599 is correct, confirming server[1] correctly received the first snapshot.

Code References

Key code paths involved:

1. getSenderReplicas (replica_command.go:2723-2821) — determines delegate vs self-send
2. IsStoreHealthy (store_pool.go:871-882) — filters candidate senders
3. followerSendSnapshot (replica_command.go:3240-3412) — records metrics on the delegate's store
4. NumDelegateLimit (replica_command.go:3192-3202) — defaults to 1, enabling delegation

Things Ruled Out

1. Lease upgrade race (prior flake): The prior issue kv/kvserver: TestRebalancingAndCrossRegionZoneSnapshotMetrics failed #137249 was about snapshot size differences due to lease type. The current failure shows bytes on the wrong server, not a size mismatch. Different root cause.
2. Raft snapshot queue: Disabled via DisableRaftSnapshotQueue = true in the test knobs.
3. Leader lease transfer: While a lease transfer to server[1] could cause server[1] to be the coordinator, the .cross-zone classification of server[1]'s sent bytes indicates the coordinator was still server[0] (us-east to us-east = cross-zone). If server[1] (us-west) were the coordinator, it would be classified as cross-region to server[2] (us-east).
4. Gossip locality race: Even if locality info were missing, the diversity score calculations would still result in the coordinator being selected (empty locality produces maximum diversity score, so the coordinator ties with others and wins via the early-return check). This only fails when the coordinator is excluded from the candidates entirely (unhealthy).

Other Failure Occurrences

Only one failure occurrence in this issue. The prior issue #137249 had a different failure mode (byte count mismatch, not wrong-server sending).

Tooling Feedback

• EngFlow artifacts were inaccessible (ECONNREFUSED when trying to reach mesolite.cluster.engflow.com, and the engflow_artifacts.py script was not present at the failure SHA and required extraction from master + manual execution which wasn't permitted). Having pre-approved access to the EngFlow script or pre-downloaded test logs would have allowed confirming the store pool health state at the time of the failure.

---

Investigation triggered by this workflow run.

Was this investigation helpful? Leave a 👍 or 👎 on this comment.