Skip to content

Commit 95efcfd

Browse files
committed
test(cluster): increase harness timeouts to reduce flaky failures
Cluster bootstrap, join, and convergence waits were capped at 10s, which is too tight under load (CI, slow VMs). Raise all cluster harness deadlines to 30s and add a matching nextest slow-timeout override so the test runner does not terminate the cluster-tests package before those waits can complete.
1 parent 02a58db commit 95efcfd

3 files changed

Lines changed: 17 additions & 10 deletions

File tree

.config/nextest.toml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,13 @@ threads-required = 'num-test-threads'
4343
# legitimate startup jitter without hiding real regressions — a
4444
# genuinely broken test fails twice in a row.
4545
retries = { backoff = "fixed", count = 2, delay = "1s" }
46+
# Cluster tests legitimately need more wall-clock than unit tests:
47+
# a 3-node cluster bring-up plus a multi-node DDL fan-out (e.g.
48+
# CREATE INDEX backfill across every peer) can stretch past the
49+
# default 120s ceiling under load without anything being wrong.
50+
# This override applies ONLY to the cluster-tests package — the
51+
# default 30s × 4 = 120s ceiling stays in force for unit tests.
52+
slow-timeout = { period = "30s", terminate-after = 6 }
4653

4754
[test-groups]
4855
cluster = { max-threads = 1 }

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -210,11 +210,11 @@ cargo nextest run --all-features
210210

211211
## Release Status
212212

213-
NodeDB Origin is in **public beta** as of **v0.1.0 (2026-05-01)**. All eight engines are feature-complete and covered by tests. The wire protocols (pgwire, HTTP, native MessagePack, RESP, ILP, WebSocket) are stable — clients written against 0.1.0 will keep working through 1.0.
213+
NodeDB Origin is in **public beta** as of **v0.1.0 (2026-05-07)**. All eight engines are feature-complete and covered by tests. The wire protocols (pgwire, HTTP, native MessagePack, RESP, ILP, WebSocket) are stable — clients written against 0.1.0 will keep working through 1.0.
214214

215215
**v0.1.0 — Beta (today).** Build new products on it. The public surface (SQL dialect, wire protocols, configuration) is stable; expect internal changes (storage layout, on-disk format, replication internals) between minor releases. Patch and minor bumps will land as needed — if something requires a 0.2, we ship a 0.2. The two months between beta and 1.0 are deliberately for real workloads to surface edge cases we can't manufacture in-house.
216216

217-
**v1.0.0 — Production-ready (target: 2026-07-01).** What 1.0 guarantees:
217+
**v1.0.0 — Production-ready (target: 2026-07-07).** What 1.0 guarantees:
218218

219219
- **API & SQL stability** — semver from 1.0 onward. No breaking SQL or client-API changes within a major.
220220
- **Wire protocol stability** — pgwire, HTTP, native MessagePack, RESP, ILP, WebSocket frozen.

nodedb-test-support/src/cluster_harness/cluster.rs

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -52,10 +52,10 @@ impl TestCluster {
5252
// short under heavy host load (e.g. 500+ parallel unit tests
5353
// sharing the same CPU pool), causing peers to dial before
5454
// node 1's transport was ready — failing topology convergence.
55-
let deadline = std::time::Instant::now() + Duration::from_secs(10);
55+
let deadline = std::time::Instant::now() + Duration::from_secs(30);
5656
while node1.topology_size() < 1 {
5757
if std::time::Instant::now() >= deadline {
58-
return Err("node 1 failed to bootstrap within 10s".into());
58+
return Err("node 1 failed to bootstrap within 30s".into());
5959
}
6060
tokio::time::sleep(Duration::from_millis(20)).await;
6161
}
@@ -67,10 +67,10 @@ impl TestCluster {
6767
// Under load, spawning both peers simultaneously can overwhelm the
6868
// bootstrap leader's join handler, causing neither join to complete
6969
// within the topology convergence deadline.
70-
let deadline = std::time::Instant::now() + Duration::from_secs(10);
70+
let deadline = std::time::Instant::now() + Duration::from_secs(30);
7171
while node1.topology_size() < 2 {
7272
if std::time::Instant::now() >= deadline {
73-
return Err("node 2 failed to join within 10s".into());
73+
return Err("node 2 failed to join within 30s".into());
7474
}
7575
tokio::time::sleep(Duration::from_millis(20)).await;
7676
}
@@ -83,7 +83,7 @@ impl TestCluster {
8383

8484
wait_for(
8585
"all 3 nodes report topology_size == 3",
86-
Duration::from_secs(10),
86+
Duration::from_secs(30),
8787
Duration::from_millis(50),
8888
|| cluster.nodes.iter().all(|n| n.topology_size() == 3),
8989
)
@@ -114,7 +114,7 @@ impl TestCluster {
114114
// replication.
115115
wait_for(
116116
"all 3 nodes exit rolling-upgrade compat mode",
117-
Duration::from_secs(10),
117+
Duration::from_secs(30),
118118
Duration::from_millis(20),
119119
|| {
120120
cluster.nodes.iter().all(|n| {
@@ -145,7 +145,7 @@ impl TestCluster {
145145
// wasted CI minutes on cleanup of a doomed cluster bringup.
146146
wait_for(
147147
"metadata group has stable leader visible on every node",
148-
Duration::from_secs(10),
148+
Duration::from_secs(30),
149149
Duration::from_millis(20),
150150
|| {
151151
let leaders: Vec<u64> = cluster
@@ -182,7 +182,7 @@ impl TestCluster {
182182
// window deterministically.
183183
wait_for(
184184
"every Raft group has a stable leader visible on every node",
185-
Duration::from_secs(10),
185+
Duration::from_secs(30),
186186
Duration::from_millis(20),
187187
|| {
188188
// Snapshot every node's per-group leader view. A group

0 commit comments

Comments
 (0)