Skip to content

Commit 4582462

Browse files
committed
test(harness): widen internal request deadline for shared CI runners
A 3-node cluster packed onto ~2 shared CI cores runs several times slower than real hardware, so the production 30s internal request deadline could spuriously abort cross-shard transactions that complete in ~10s locally. Raise it for harness-spawned nodes only, staying below nextest's cluster terminate ceiling so a genuine hang still fails cleanly.
1 parent 568c558 commit 4582462

1 file changed

Lines changed: 24 additions & 0 deletions

File tree

  • nodedb-test-support/src/cluster_harness/node/lifecycle

nodedb-test-support/src/cluster_harness/node/lifecycle/spawn_full.rs

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,15 @@ use crate::cluster_harness::cluster::ClusterSpawnConfig;
2424
use super::client_slot::ClusterTestClient;
2525
use super::types::{DataDir, TestClusterNode};
2626

27+
/// Internal request deadline for harness-spawned nodes, widened well above the
28+
/// production 30s default. Cross-shard work on a CI runner that packs a 3-node
29+
/// cluster onto ~2 shared cores runs several times slower than on real
30+
/// hardware, so the production deadline spuriously aborts transactions that
31+
/// complete in ~10s locally. This value sits above the slowest observed
32+
/// cross-node operation yet below nextest's cluster terminate ceiling, so a
33+
/// genuine hang still fails cleanly instead of being masked.
34+
const TEST_REQUEST_DEADLINE_SECS: u64 = 120;
35+
2736
impl TestClusterNode {
2837
/// [`Self::spawn_with_full_config`] entry point used by every spawn
2938
/// variant that does not need to reopen an existing data directory: mints
@@ -165,6 +174,21 @@ impl TestClusterNode {
165174
// Wire cluster handles into SharedState (mirrors main.rs).
166175
// `Arc::get_mut` is valid here: `shared` has not been cloned.
167176
if let Some(state) = Arc::get_mut(&mut shared) {
177+
// Widen the internal request deadline for the test environment. In
178+
// production a node owns real cores; here a 3-node cluster is packed
179+
// onto a CI runner with as few as 2 shared cores (one Data-Plane
180+
// core per node, plus each node's Tokio/Raft), so a heavy cross-shard
181+
// transaction — e.g. an OLLP predicate delete that resolves a
182+
// surrogate per matched edge across nodes — runs several times slower
183+
// than on real hardware and can exceed the production 30s deadline
184+
// even though it completes in ~10s locally. This is the same class of
185+
// timing compensation the harness already applies via fast-election
186+
// `ClusterTransportTuning`: the internal request deadline is the one
187+
// knob that was still left at its production default. It only ever
188+
// GRANTS headroom, so it cannot hide a regression — a genuine hang
189+
// is still terminated by nextest's cluster ceiling (see
190+
// `.config/nextest.toml`), which is lower than this deadline.
191+
state.tuning.network.default_deadline_secs = TEST_REQUEST_DEADLINE_SECS;
168192
state.node_id = handle.node_id;
169193
state.cluster_topology = Some(Arc::clone(&handle.topology));
170194
state.cluster_routing = Some(Arc::clone(&handle.routing));

0 commit comments

Comments
 (0)