roachtest: kv/splits/nodes=3/quiesce=true/lease=epoch failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.kv/splits/nodes=3/quiesce=true/lease=epoch failed with artifacts on release-26.1 @ be2205a28e0acd291fb05a5e9909d4d71afca036:

Failed with:

(cluster.go:2518).Run: full command output in run_115643.810276887_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv/splits/nodes=3/quiesce=true/lease=epoch/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21350034-1778913376-147-n4cpu4-0001	34.75.227.17	10.142.2.111
teamcity-21350034-1778913376-147-n4cpu4-0002	34.75.44.125	10.142.1.232
teamcity-21350034-1778913376-147-n4cpu4-0003	34.24.11.227	10.142.2.115
teamcity-21350034-1778913376-147-n4cpu4-0004	104.196.219.26	10.142.0.108

Parameters:

• MetamorphicFilesystem=ext4
• MetamorphicVolumeType=local-ssd
• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: kv/splits/nodes=3/quiesce=true/lease=epoch failed #170041 roachtest: kv/splits/nodes=3/quiesce=true/lease=epoch failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv branch-release-25.2]
• roachtest: kv/splits/nodes=3/quiesce=true/lease=epoch failed #169319 roachtest: kv/splits/nodes=3/quiesce=true/lease=epoch failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-25.2.18-rc]
• roachtest: kv/splits/nodes=3/quiesce=true/lease=epoch failed #168374 roachtest: kv/splits/nodes=3/quiesce=true/lease=epoch failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.3]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-64000

[pav-kv]

/investigate

[github-actions]

Investigation: kv/splits/nodes=3/quiesce=true/lease=epoch

Investigated failure: #170437
Failure SHA: be2205a28e0acd291fb05a5e9909d4d71afca036
Confidence: high

What This Test Does

The kv/splits/nodes=3/quiesce=true/lease=epoch roachtest creates a 3-node cluster (with a 4th workload node) and runs cockroach workload run kv --init --max-ops=1 --concurrency=192 --splits=300000 to perform 300,000 range splits. It verifies the cluster can handle a very high number of ranges (~100k per node) with quiescence enabled and epoch-based leases, within a 2-hour timeout. The test code explicitly documents that this test "sometimes fails since it's pushing far past the number of replicas per node we support."

Where the Failure Occurs

The workload failed at 214,000 of 300,000 splits (~71% complete) with a circuit breaker / slow proposal error on n3:

Error: executing ALTER TABLE kv SPLIT AT VALUES (2936865589849610104): pq: replica unavailable:
(n3,s3):3 unable to serve request to r125235: closed timestamp: 1778934354.750435855,0
(2026-05-16 12:25:54); raft status: {"id":"3","term":33,"vote":"3","commit":43,"lead":"2",
"raftState":"StateFollower","applied":41}: have been waiting 61.50s for slow proposal
EndTxn(commit split) [txn: e567dc7b]

The circuit breaker on n3 tripped first for r61743 at 12:26:09 due to a slow GC proposal (waiting 61s), then for r125235 at 12:27:22 due to a slow EndTxn(commit split) proposal (waiting 61.5s). After the breaker tripped, latches were poisoned on r125235, blocking the HeartbeatTxn RPC (stuck 86.34s), which surfaced to the workload as the fatal error.

The error originates at replica_circuit_breaker.go:163, triggered by slow proposal detection at replica_raft.go:1672.

Analysis

Hypothesis: Resource exhaustion under extreme range count causes replica unavailability and circuit breaker trips (likely cause, high confidence)

This is the same well-known failure mode seen across many prior occurrences of this test. The evidence:

1. CPU saturation: All nodes were running at 375-383% CPU on 4-vCPU machines (94-96% utilization). At failure time, n3 showed 379.9/7.3 %(u/s)time with 8.4 GiB RSS, n1 showed 382.7/5.1 %(u/s)time with 8.4 GiB RSS, n2 showed 378.8/7.6 %(u/s)time with 8.5 GiB RSS.

2. Memory pressure: All nodes had ~8.4-8.5 GiB RSS with Go allocating 6.3-7.0 GiB against ~8.0-8.1 GiB total.

3. Raft overload: n3 logged 632 "node might be overloaded" warnings, n2 logged 120, n1 logged 54. Raft ready handling times reached 3.38s (n3, r47828 at 12:09:14) and 5.00s (n3, r160705 at 12:17:47).

4. Replication lag: n3 logged replication lags of 1-3.5s for lease applications starting at 12:10:17.

5. Split rate degradation shows progressive overload:

   • 1-100k splits: ~7.6s per 1000 splits (steady)
   • 100-200k: ~7.8-8.5s per 1000 splits (slight slowdown)
   • 200-210k: ~9.3-14.2s per 1000 splits (degrading)
   • 210-214k: ~16.1-20.8s per 1000 splits (severe)
   • At 214k: fatal slow proposal error

6. Circuit breaker cascade: The first breaker trip was on r61743 for a slow GC proposal at 12:26:09, followed by the split proposal on r125235 at 12:27:22. The poisoned latches then blocked HeartbeatTxn RPCs, causing the workload to fail.

Notable difference from prior failures: This failure occurred earlier (214k splits, 71%) compared to prior occurrences (typically 276k-299k, 92-99%). Two factors may explain this:

• localSSD=true: This run used local SSDs (prior failures used pd-ssd). While local SSDs are typically faster, they have different I/O characteristics.
• metamorphicWriteBuffering=true: This run had write buffering enabled. Write buffering can increase memory pressure and change the timing of batch writes, potentially exacerbating resource exhaustion earlier.
• state_assertions=1 in raft ready logs indicates runtime assertion checks were active, adding overhead.

Despite the earlier failure point, the mechanism is identical to all prior occurrences: progressive resource exhaustion → slow raft proposals → circuit breaker trip → poisoned latch cascade.

Related Issues and PRs

This is a duplicate of the well-established pattern tracked across these issues:

• #166202 — Open, canonical tracking issue, same test on release-26.2 (P-3)
• #170041 — Open, same test on release-25.2, same root cause (299k splits, prior /investigate confirms identical pattern)
• #170529 — Open, same test, likely same root cause
• #169319 — Open, same test on release-25.2.18-rc, same root cause (294k splits, marked X-duplicate)
• #168374 — Open, same test on release-24.3, same root cause (276k splits, marked X-duplicate)

All share the same root cause: resource exhaustion at high range counts on 4-vCPU machines. The failure point and surface error vary depending on machine conditions.

Timeline

Time (UTC)	Event
11:56:43	Workload starts: 300k splits with 192 concurrency
11:56:53	1k splits (7.8s per batch — normal)
12:09:14	First "node might be overloaded" on n3 (r47828, 3.38s raft ready) at ~97k splits
12:09:46	100k splits (~7.6s per batch — still OK)
12:10:17	First replication lag warning on n3 (~3.46s lag)
12:17:24-47	Severe raft ready delays on n3 (1.5-5.0s) at ~160k splits
12:23:28	200k splits (~12.5s per batch — degrading)
12:25:09	~209k splits (~15.4s per batch — accelerating degradation)
~12:25:09	r61743 GC proposal stalls on n3 (closed timestamp 12:24:46)
12:26:09	Slow GC proposal exceeds 61s on r61743, circuit breaker trips on n3
12:27:05	214k splits — last successful batch (~20.8s per batch)
12:27:22	Slow split proposal exceeds 61.5s on r125235, circuit breaker trips
12:27:23	Poisoned latch cascade, HeartbeatTxn stuck 86.34s
12:27:41	Workload fails with COMMAND_PROBLEM: exit status 1

Detailed evidence and investigation notes

Log Excerpts

First slow proposal — GC on r61743 (n3, 12:26:09):

W260516 12:26:09.999229 [n3,s3,r61743/3] have been waiting 61.00s for slow proposal
GC [/Table/106/1/...]

Circuit breaker trip on r61743 (n3, 12:26:09):

E260516 12:26:09.999684 [n3,s3,r61743/3] breaker: tripped with error: replica unavailable:
(n3,s3):3 unable to serve request to r61743: closed timestamp: 2026-05-16 12:24:46;
raft status: {"id":"3","term":27,"vote":"2","commit":45,"lead":"2",
"raftState":"StateFollower","applied":44}: have been waiting 61.00s for slow proposal GC

Fatal slow split proposal on r125235 (n3, 12:27:22):

W260516 12:27:22.985817 [n3,s3,r125235/3] have been waiting 61.50s for slow proposal
EndTxn(commit split) [...], [txn: e567dc7b]

Poisoned latch — HeartbeatTxn stuck 86.34s (n3, 12:27:23):

W260516 12:27:23.014580 [n3,txn-hb=e567dc7b] slow replica RPC: have been waiting 86.34s
for RPC HeartbeatTxn to replica (n3,s3):3; resp: encountered poisoned latch
/Local/Range/Table/106/1/.../RangeDescriptor@0,0

n3 runtime stats at failure time:

12:26:44 runtime stats: 8.4 GiB RSS, 2253 goroutines, 6.3 GiB/8.0 GiB Go alloc/total,
  379.9/7.3 %(u/s)time
12:27:04 runtime stats: 8.4 GiB RSS, 2234 goroutines, 6.6 GiB/8.0 GiB Go alloc/total,
  377.9/7.5 %(u/s)time

Raft overload representative examples on n3:

12:09:14 r47828: raft ready handling: 3.38s [apply=3.38s]; state_assertions=1; node might be overloaded
12:17:47 r160705: raft ready handling: 5.00s [other=4.99s]; node might be overloaded
12:18:24 r101815: raft ready handling: 3.18s [other=3.18s]; node might be overloaded

Code References

• Test registration: kv.go:681-772
• Fragility caveat: kv.go:689-692
• Slow proposal detection: replica_raft.go:1672
• Circuit breaker trip: replica_circuit_breaker.go:163

Things Ruled Out

• OOM kill: No OOM evidence in dmesg on any node. All nodes remained running.
• Node crash: All 3 CockroachDB nodes remained running throughout the test.
• Code regression: Same failure pattern has been occurring across release-24.3, release-25.2, release-26.1, and release-26.2 for months. Not tied to a specific recent commit.
• Network partition: All nodes were communicating; the issue was slow raft application, not network failure.
• Branch-specific issue: This exact failure has been reproduced on multiple branches.

Other Failure Occurrences

Issue	Branch	Splits at Failure	Error Type
#170437 (this)	release-26.1	214k/300k	slow GC proposal + split proposal
#170041	release-25.2	299k/300k	slow split proposal
#169319	release-25.2.18-rc	294k/300k	poisoned latch
#168374	release-24.3	276k/300k	poisoned latch
#166202	release-26.2	174k/300k	acquire-lease timeout

All share the same root cause. This occurrence failed earlier than most (214k vs typical 276k-299k), possibly due to metamorphicWriteBuffering=true and localSSD=true metamorphic parameters adding overhead.

Tooling Feedback

No tooling gaps encountered during this investigation. All necessary artifacts and logs were accessible via TeamCity guest authentication.

---

Workflow run

---

Was this investigation helpful? Leave a 👍 or 👎 on this comment.