roachtest: add splits perturbation test

Reframe "how many splits can the cluster sustain" as a perturbation
test that runs concurrently with a foreground KV workload and reports
impact in roachperf-comparable terms (throughput ratio, density in
replicas/vcpu/node). Supersedes the legacy kv/splits/*/quiesce=*/lease=*
binary tests dropped earlier in this PR.

Two flavors are registered together: asleep=true measures total replica
capacity (followers park via store liveness quiescence), asleep=false
measures active-replica capacity (every replica costs full per-tick
CPU, the relevant ceiling during outages where followers can't sleep).
asleep=true targets ~9,400 replicas/vcpu (40% over the production
telemetry cluster's ~6,700 replicas/vcpu non-quiescent density) to
stress-test the primary quiescence path beyond current production
density; asleep=false targets ~6,250 replicas/vcpu to match the
telemetry cluster's observed non-quiescent density. asleep=false also
disables store-liveness follower sleep via the
kv.raft.store_liveness.quiescence.enabled cluster setting; leader
leases keep leaseholders awake unconditionally so no additional knob
is needed.

Design (see the struct-level comment in splits.go for the long form):

- Partition the cluster's stores into disjoint groups of three on
  distinct nodes; pin one database per group with voter_constraints
  that one-to-one map each voter to a store. With this every split's
  children inherit their parent's correct placement, so no SCATTER and
  no rebalance churn during the split storm.
- Issue splits per group in parallel batches; descending keys per batch
  to keep each split landing on the leftmost (stable, single-leader)
  range rather than chaining new raft groups together.
- After batch 0 (and again at end-of-perturbation), do an explicit
  deterministic round-robin lease rebalance via ALTER RANGE RELOCATE
  LEASE. Plain SCATTER stalls at 1.5-3x spread under per-store
  voter_constraints because the allocator's lease scorer picks the
  same biased target on every re-scatter. Round-robin converges in one
  pass; dispatch is parallelized within a group (concurrency 16) and
  across groups (one task per group).
- Per-call ALTER RANGE failures are tolerated up to 10% (catches
  systemic problems like a node down while shrugging off transient
  range-merge races); a 100%-failure floor catches small-batch
  pathologies that the percentage gate would skip. First error is
  surfaced in the failure message and in a warn log when any failure
  happens below the threshold.

Knobs in splits.setup():
- asleep=true: targetSplits=600k, batchSize=10k (full); the optional
  devSizer / metamorphicSizer interfaces (added in this commit) let
  the dev and metamorphic variants shrink to 4k and 60k respectively.
- asleep=false: targetSplits=400k, batchSize=5k (full); 40k metamorphic;
  shares the 4k dev with asleep=true.
- v.splits=500 + v.scatter=true: enough kv ranges to avoid a
  single-range hotspot for the foreground workload, scattered so
  leases distribute evenly.
- v.timeout=5h: full 600k storm + end-of-perturbation rebalance runs
  for over an hour; the default 3h budget is too tight. asleep=false
  shares the budget since the rebalance phase dominates either way.
- v.tokenReturnTime=1h: real outstanding flow-control tokens drain
  within seconds of workload cancel, but ValidateTokensReturned's SQL
  query races against background flow-control activity (replicate
  queue, lease queue, jobs) at 1-2M replicas and observed transient
  non-zero diffs at the 10m default. A longer window gives the retry
  loop a better chance to land on a clean snapshot.
- impact=10x, recoveryImpact=3x: during the split storm the cluster is
  expected to absorb significant throughput loss; the meaningful
  pass/fail signal is whether it returns to baseline afterwards. Both
  flavors share these thresholds; the asleep=false gates may be
  tightened in a follow-up now that the flavor lands at production
  density.
- SkipPostValidations=PostValidationReplicaDivergence: at 1.2M-1.8M
  ranges the post-test consistency check can't scan a meaningful
  fraction within the framework's 20m budget (observed: 0 ranges in
  20m). The framework tolerates the timeout but logs it as a
  post-assertion failure; skipping explicitly keeps the artifact clean.

addDev now also zeros v.recoveryImpact, v.timeout, and v.tokenReturnTime,
all of which splits.setup() sets to non-zero values. Without these
clears the dev variant would inherit a 3x recovery throughput gate
(too tight for noisy local processes), a 5h test timeout (would hide
hangs behind a long wait), and a 1h token-return wait (wildly
overshoots what's meaningful on a 5-node local cluster).

Test fixtures:
- waitForVoterPlacement polls per-range placement against the group's
  expected store set and nudges non-conforming ranges into the
  replicate queue via crdb_internal.kv_enqueue_replica (must run on
  the leaseholder).
- waitForVoterPlacement budget defaults to 10 minutes and scales up to
  v.timeout/3 (capped at 90 minutes) when the test sets a per-test
  timeout. The 10-minute default suits the dev variant; the lift makes
  the nightly variant tolerant of slow up-replication.
- reportDensity / reportBalance / summarize give per-node and per-store
  range and lease distribution at every batch boundary, with
  deterministic min/max tiebreaks so log diffs are stable across runs.

Epic: none
Release note: None

Author: tbg

pkg/cmd/roachtest/tests/perturbation/BUILD.bazel

@@ -14,6 +14,7 @@ go_library(
         "network_partition.go",
         "restart_node.go",
         "slow_disk.go",
+        "splits.go",
     ],
     importpath = "github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/perturbation",
     visibility = ["//visibility:public"],
@@ -23,6 +24,7 @@ go_library(
         "//pkg/cmd/roachtest/registry",
         "//pkg/cmd/roachtest/roachtestutil",
         "//pkg/cmd/roachtest/roachtestutil/artifactsutil",
+        "//pkg/cmd/roachtest/roachtestutil/task",
         "//pkg/cmd/roachtest/spec",
         "//pkg/cmd/roachtest/test",
         "//pkg/roachprod/install",
@@ -34,6 +36,7 @@ go_library(
         "//pkg/workload/cli",
         "//pkg/workload/histogram",
         "@com_github_cockroachdb_errors//:errors",
+        "@com_github_lib_pq//:pq",
         "@com_github_stretchr_testify//require",
     ],
 )

pkg/cmd/roachtest/tests/perturbation/framework.go

@@ -123,6 +123,11 @@ type variations struct {
 	profileOptions     []roachtestutil.ProfileOptionFunc
 	specOptions        []spec.Option
 	clusterSettings    map[string]string
+	// skipPostValidations is forwarded to TestSpec.SkipPostValidations. The
+	// zero value runs all post-test validations. Perturbations that
+	// inflate the cluster (large range counts, heavy fills) can set this
+	// to skip checks that exceed the framework's per-check budget.
+	skipPostValidations registry.PostValidation
 }
 
 const NUM_REGIONS = 3
@@ -395,6 +400,9 @@ func setup(p perturbation, impact acceptableImpact) variations {
 		roachtestutil.ProfMultipleFromP99(10),
 	}
 	v.impact = impact
+	// recoveryImpact left at zero value — falls back to impact at the
+	// runTest use site. Tests that want a different threshold for the
+	// recovery interval set it explicitly.
 	v.clusterSettings = make(map[string]string)
 	// Having the io_load_listener logs makes it easier to debug failures.
 	v.clusterSettings["server.debug.default_vmodule"] = "io_load_listener=1"
@@ -416,6 +424,13 @@ func RegisterTests(r registry.Registry) {
 	register(r, restart{}, notSkipped)
 	addLong(r, restart{})
 	register(r, backup{}, notSkipped)
+	// splits runs in two flavors: asleep=true exercises total replica
+	// capacity (followers park via store liveness quiescence), asleep=false
+	// exercises active-replica capacity (every replica costs full per-tick
+	// CPU). See splits.go for the long-form rationale.
+	for _, asleep := range []bool{true, false} {
+		register(r, splits{asleep: asleep}, notSkipped)
+	}
 
 	// TODO(ssd): We skipped the majority of these tests so that we can focus on
 	// one at a time. These are vaguely ordered by their previous pass rate
@@ -571,6 +586,13 @@ func addMetamorphic(r registry.Registry, p perturbation, skipReason string) {
 	rng, seed := randutil.NewPseudoRand()
 	v := p.setupMetamorphic(rng)
 	v.seed = seed
+	// Some perturbations need to scale down for the metamorphic variant
+	// (the randomized clusters this suite picks are smaller than the
+	// dedicated nightly cluster). They implement metamorphicSizer to
+	// return a shrunk version of themselves.
+	if ms, ok := v.perturbation.(metamorphicSizer); ok {
+		v.perturbation = ms.sizeForMetamorphic()
+	}
 	v = v.finishSetup()
 	spec := registry.TestSpec{
 		Name:                   fmt.Sprintf("perturbation/metamorphic/%s", v.perturbationName()),
@@ -581,6 +603,7 @@ func addMetamorphic(r registry.Registry, p perturbation, skipReason string) {
 		Cluster:                v.makeClusterSpec(),
 		Leases:                 v.leaseType,
 		Randomized:             true,
+		SkipPostValidations:    v.skipPostValidations,
 		PostProcessPerfMetrics: perturbationDefaultProcessFunction,
 		Run:                    v.runTest,
 	}
@@ -600,6 +623,7 @@ func addFull(r registry.Registry, p perturbation, skipReason string) {
 		Cluster:                v.makeClusterSpec(),
 		Leases:                 v.leaseType,
 		Benchmark:              true,
+		SkipPostValidations:    v.skipPostValidations,
 		PostProcessPerfMetrics: perturbationDefaultProcessFunction,
 		Run:                    v.runTest,
 	}
@@ -673,8 +697,22 @@ func addLong(r registry.Registry, p perturbation) {
 
 func addDev(r registry.Registry, p perturbation) {
 	v := p.setup()
-	// Dev tests never fail on latency increases.
+	// Dev tests never fail on latency or throughput changes — local
+	// processes on a developer machine are too noisy for either to be
+	// meaningful. Zero out both intervals' thresholds. recoveryImpact
+	// must be zeroed explicitly even though impact is also zero: a
+	// perturbation setup that set recoveryImpact to a non-zero value
+	// would otherwise survive and gate the dev run.
 	v.impact = noImpactThresholds()
+	v.recoveryImpact = acceptableImpact{}
+	// Clear any per-test timeout override; the registry default is fine
+	// for a 30-second perturbation on a 5-node local cluster, and an
+	// override sized for the nightly variant (e.g. several hours) would
+	// hide hangs in the dev run. Same reasoning for tokenReturnTime:
+	// the nightly override (e.g. 1h) wildly overshoots what's
+	// meaningful on a 5-node local cluster.
+	v.timeout = 0
+	v.tokenReturnTime = 0
 	// Make the tests faster for development.
 	v.splits = 1
 	v.numNodes = 5
@@ -696,6 +734,12 @@ func addDev(r registry.Registry, p perturbation) {
 
 	// Allow the test to run on dev machines.
 	v.cloud = registry.AllClouds
+	// Some perturbations need to scale down for the dev variant (smaller
+	// cluster, faster wall time). They implement devSizer to return a
+	// shrunk version of themselves.
+	if ds, ok := v.perturbation.(devSizer); ok {
+		v.perturbation = ds.sizeForDev()
+	}
 	v = v.finishSetup()
 	spec := registry.TestSpec{
 		Name:                   fmt.Sprintf("perturbation/dev/%s", v.perturbationName()),
@@ -747,6 +791,18 @@ type perturbationNamer interface {
 	nameSuffix() string
 }
 
+// devSizer is an optional interface for perturbations that need to shrink
+// themselves for the dev variant (small cluster, fast wall time). addDev
+// invokes sizeForDev() if the perturbation implements it.
+type devSizer interface {
+	sizeForDev() perturbation
+}
+
+// metamorphicSizer is the same idea as devSizer for the metamorphic variant.
+type metamorphicSizer interface {
+	sizeForMetamorphic() perturbation
+}
+
 func prettyPrint(title string, stats map[string]aggregatedStat) string {
 	var outputStr strings.Builder
 	outputStr.WriteString(title + "\n")