roachtest/perturbation: wait for rebalance after baseline; scatter restart init

Two changes motivated by the recovery-phase write stall analyzed in
#170849.

  - In the shared framework runTest, wait for in-flight rebalancing
    to settle after the baseline interval and before T3. This applies
    to all perturbation tests; the wait is a no-op when nothing is
    rebalancing, so non-restart tests only pay the polling cost. On
    restart it closes the per-store range-count skew left by the fill
    phase, which otherwise concentrates the recovery snapshot storm
    on whichever of the target node's stores looks most underfull.

  - Pass --scatter to `kv workload init` for restart so the workload
    table's initial splits are distributed across stores rather than
    landing wherever the allocator happens to pick at split time.

Release note: None

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Author: miraradeva

pkg/cmd/roachtest/tests/perturbation/framework.go

@@ -992,6 +992,14 @@ func (v variations) runTest(ctx context.Context, t test.Test, c cluster.Cluster)
 
 	// Collect the baseline after the workload has stabilized.
 	baselineInterval := intervalSince(v.validationDuration / 2)
+
+	// Let any in-flight rebalancing settle before perturbing. The fill phase
+	// can leave per-store range counts noticeably skewed: when the perturbed
+	// node returns and the cluster tries to re-balance to its underfull
+	// stores, the snapshot storm concentrates on one store and can push it
+	// into a Pebble write stall.
+	v.waitForRebalanceToStop(ctx, t)
+
 	// Now start the perturbation.
 	t.Status("T3: inducing perturbation")
 	perturbationDuration := v.perturbation.startPerturbation(ctx, t, v)

pkg/cmd/roachtest/tests/perturbation/restart_node.go

@@ -34,6 +34,12 @@ func (r restart) setup() variations {
 	// flush/compaction pipeline can fall behind into a Pebble write stall.
 	v.ratioOfMax = 0.3
 
+	// Scatter the workload table at init so initial replica placement isn't
+	// skewed across the per-node stores. A pre-existing per-store imbalance
+	// concentrates the recovery snapshot storm on whichever store appears
+	// underfull, which is what causes the stall described above.
+	v.scatter = true
+
 	// TODO(baptist): Remove this setting once #120073 is fixed.
 	v.clusterSettings["kv.lease.reject_on_leader_unknown.enabled"] = "true"