roachtest: add splits perturbation test

Reframe "how many splits can the cluster sustain" as a perturbation
test that runs concurrently with a foreground KV workload and reports
impact in roachperf-comparable terms (throughput ratio, density in
replicas/vcpu/node). Supersedes the legacy kv/splits/*/quiesce=*/lease=*
binary tests dropped earlier in this PR.

Two flavors are registered together: asleep=true measures total replica
capacity (followers park via store liveness quiescence), asleep=false
measures active-replica capacity (every replica costs full per-tick
CPU, the relevant ceiling during outages where followers can't sleep).
Because awake replicas are far more expensive, the asleep=false
population is scaled down 5× at every cluster size. asleep=false also
disables store-liveness follower sleep via the
kv.raft.store_liveness.quiescence.enabled cluster setting; leader
leases keep leaseholders awake unconditionally so no additional knob
is needed.

Design (see the struct-level comment in splits.go for the long form):

- Partition the cluster's stores into disjoint groups of three on
  distinct nodes; pin one database per group with voter_constraints
  that one-to-one map each voter to a store. With this every split's
  children inherit their parent's correct placement, so no SCATTER and
  no rebalance churn during the split storm.
- Issue splits per group in parallel batches; descending keys per batch
  to keep each split landing on the leftmost (stable, single-leader)
  range rather than chaining new raft groups together.
- After batch 0 (and again at end-of-perturbation), do an explicit
  deterministic round-robin lease rebalance via ALTER RANGE RELOCATE
  LEASE. Plain SCATTER stalls at 1.5-3x spread under per-store
  voter_constraints because the allocator's lease scorer picks the
  same biased target on every re-scatter. Round-robin converges in one
  pass; dispatch is parallelized within a group (concurrency 16) and
  across groups (one task per group).
- Per-call ALTER RANGE failures are tolerated up to 10% (catches
  systemic problems like a node down while shrugging off transient
  range-merge races); a 100%-failure floor catches small-batch
  pathologies that the percentage gate would skip. First error is
  surfaced in the failure message and in a warn log when any failure
  happens below the threshold.

Knobs in splits.setup():
- targetSplits=600k, batchSize=10k (asleep=true, full variant); the
  optional devSizer / metamorphicSizer interfaces (added in this
  commit) let the dev and metamorphic variants shrink to 4k and 60k
  respectively. asleep=false uses 120k/2k/12k/800 at full/metamorphic/
  dev (5× smaller across the board).
- v.splits=500 + v.scatter=true: enough kv ranges to avoid a
  single-range hotspot for the foreground workload, scattered so
  leases distribute evenly.
- v.timeout=5h: full 600k storm + end-of-perturbation rebalance runs
  for over an hour; the default 3h budget is too tight. asleep=false
  shares the budget since the rebalance phase dominates either way.
- v.tokenReturnTime=1h: real outstanding flow-control tokens drain
  within seconds of workload cancel, but ValidateTokensReturned's SQL
  query races against background flow-control activity (replicate
  queue, lease queue, jobs) at 1.8M replicas and observed transient
  non-zero diffs at the 10m default. A longer window gives the retry
  loop a better chance to land on a clean snapshot.
- impact=10x, recoveryImpact=3x: during the split storm the cluster is
  expected to absorb significant throughput loss; the meaningful
  pass/fail signal is whether it returns to baseline afterwards.
  asleep=false reuses these thresholds for now; they may be tightened
  after the first calibration run if the lower replica population
  produces materially less impact.

addDev now also zeros v.recoveryImpact and v.timeout, both of which
splits.setup() sets to non-zero values. Without these clears the dev
variant would inherit a 3x recovery throughput gate (too tight for
noisy local processes) and a 5h test timeout (would hide hangs behind
a long wait).

Test fixtures:
- waitForVoterPlacement polls per-range placement against the group's
  expected store set and nudges non-conforming ranges into the
  replicate queue via crdb_internal.kv_enqueue_replica (must run on
  the leaseholder).
- waitForVoterPlacement budget defaults to 10 minutes and scales up to
  v.timeout/3 (capped at 90 minutes) when the test sets a per-test
  timeout. The 10-minute default suits the dev variant; the lift makes
  the nightly variant tolerant of slow up-replication.
- reportDensity / reportBalance / summarize give per-node and per-store
  range and lease distribution at every batch boundary, with
  deterministic min/max tiebreaks so log diffs are stable across runs.

Epic: none
Release note: None

Author: tbg

pkg/cmd/roachtest/tests/perturbation/BUILD.bazel

@@ -14,6 +14,7 @@ go_library(
         "network_partition.go",
         "restart_node.go",
         "slow_disk.go",
+        "splits.go",
     ],
     importpath = "github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/perturbation",
     visibility = ["//visibility:public"],
@@ -23,6 +24,7 @@ go_library(
         "//pkg/cmd/roachtest/registry",
         "//pkg/cmd/roachtest/roachtestutil",
         "//pkg/cmd/roachtest/roachtestutil/artifactsutil",
+        "//pkg/cmd/roachtest/roachtestutil/task",
         "//pkg/cmd/roachtest/spec",
         "//pkg/cmd/roachtest/test",
         "//pkg/roachprod/install",
@@ -34,6 +36,7 @@ go_library(
         "//pkg/workload/cli",
         "//pkg/workload/histogram",
         "@com_github_cockroachdb_errors//:errors",
+        "@com_github_lib_pq//:pq",
         "@com_github_stretchr_testify//require",
     ],
 )

pkg/cmd/roachtest/tests/perturbation/framework.go

@@ -395,6 +395,9 @@ func setup(p perturbation, impact acceptableImpact) variations {
 		roachtestutil.ProfMultipleFromP99(10),
 	}
 	v.impact = impact
+	// recoveryImpact left at zero value — falls back to impact at the
+	// runTest use site. Tests that want a different threshold for the
+	// recovery interval set it explicitly.
 	v.clusterSettings = make(map[string]string)
 	// Having the io_load_listener logs makes it easier to debug failures.
 	v.clusterSettings["server.debug.default_vmodule"] = "io_load_listener=1"
@@ -416,6 +419,13 @@ func RegisterTests(r registry.Registry) {
 	register(r, restart{}, notSkipped)
 	addLong(r, restart{})
 	register(r, backup{}, notSkipped)
+	// splits runs in two flavors: asleep=true exercises total replica
+	// capacity (followers park via store liveness quiescence), asleep=false
+	// exercises active-replica capacity (every replica costs full per-tick
+	// CPU). See splits.go for the long-form rationale.
+	for _, asleep := range []bool{true, false} {
+		register(r, splits{asleep: asleep}, notSkipped)
+	}
 
 	// TODO(ssd): We skipped the majority of these tests so that we can focus on
 	// one at a time. These are vaguely ordered by their previous pass rate
@@ -571,6 +581,13 @@ func addMetamorphic(r registry.Registry, p perturbation, skipReason string) {
 	rng, seed := randutil.NewPseudoRand()
 	v := p.setupMetamorphic(rng)
 	v.seed = seed
+	// Some perturbations need to scale down for the metamorphic variant
+	// (the randomized clusters this suite picks are smaller than the
+	// dedicated nightly cluster). They implement metamorphicSizer to
+	// return a shrunk version of themselves.
+	if ms, ok := v.perturbation.(metamorphicSizer); ok {
+		v.perturbation = ms.sizeForMetamorphic()
+	}
 	v = v.finishSetup()
 	spec := registry.TestSpec{
 		Name:                   fmt.Sprintf("perturbation/metamorphic/%s", v.perturbationName()),
@@ -670,8 +687,19 @@ func addLong(r registry.Registry, p perturbation) {
 
 func addDev(r registry.Registry, p perturbation) {
 	v := p.setup()
-	// Dev tests never fail on latency increases.
+	// Dev tests never fail on latency or throughput changes — local
+	// processes on a developer machine are too noisy for either to be
+	// meaningful. Zero out both intervals' thresholds. recoveryImpact
+	// must be zeroed explicitly even though impact is also zero: a
+	// perturbation setup that set recoveryImpact to a non-zero value
+	// would otherwise survive and gate the dev run.
 	v.impact = noImpactThresholds()
+	v.recoveryImpact = acceptableImpact{}
+	// Clear any per-test timeout override; the registry default is fine
+	// for a 30-second perturbation on a 5-node local cluster, and an
+	// override sized for the nightly variant (e.g. several hours) would
+	// hide hangs in the dev run.
+	v.timeout = 0
 	// Make the tests faster for development.
 	v.splits = 1
 	v.numNodes = 5
@@ -693,6 +721,12 @@ func addDev(r registry.Registry, p perturbation) {
 
 	// Allow the test to run on dev machines.
 	v.cloud = registry.AllClouds
+	// Some perturbations need to scale down for the dev variant (smaller
+	// cluster, faster wall time). They implement devSizer to return a
+	// shrunk version of themselves.
+	if ds, ok := v.perturbation.(devSizer); ok {
+		v.perturbation = ds.sizeForDev()
+	}
 	v = v.finishSetup()
 	spec := registry.TestSpec{
 		Name:                   fmt.Sprintf("perturbation/dev/%s", v.perturbationName()),
@@ -744,6 +778,18 @@ type perturbationNamer interface {
 	nameSuffix() string
 }
 
+// devSizer is an optional interface for perturbations that need to shrink
+// themselves for the dev variant (small cluster, fast wall time). addDev
+// invokes sizeForDev() if the perturbation implements it.
+type devSizer interface {
+	sizeForDev() perturbation
+}
+
+// metamorphicSizer is the same idea as devSizer for the metamorphic variant.
+type metamorphicSizer interface {
+	sizeForMetamorphic() perturbation
+}
+
 func prettyPrint(title string, stats map[string]aggregatedStat) string {
 	var outputStr strings.Builder
 	outputStr.WriteString(title + "\n")