docs(design): propose hotspot split M3 — automation (#951)

Design-doc-only PR. No implementation code. Per the repo's
design-doc-first workflow, this lands the M3 proposal for review before
any implementation.

New file:
`docs/design/2026_06_11_proposed_hotspot_split_milestone3_automation.md`
(`Status: Proposed`, `Author: bootjp`, `Date: 2026-06-11`).

## Summary

Milestone 3 of the [hotspot shard
split](docs/design/2026_02_18_partial_hotspot_shard_split.md) design:
**automation** — auto-detection + an auto-split scheduler. M1 (durable
versioned route catalog, manual same-group `SplitRange`, route watcher)
has shipped; [M2 (#945)](#945)
proposes the cross-group migration plane. M3 composes with M2 but
delivers first value standalone: auto-detection + same-group auto-split
(the M1 capability) work without M2; cross-group target selection slots
in behind the same `SplitRange` interface once M2 lands.

It closes the gap the parent doc §1 calls out:
`distribution/engine.go`'s `RecordAccess`/threshold-`splitRange` exists
but is wired into **nothing** (only tests call it; verified by grep) and
is midpoint-only + mutates the engine directly (bypassing the
catalog/watcher).

## Key design points

- **Detection signal — reuse the keyviz sampler, retire the engine
counters.** The keyviz `MemSampler` is already wired into the exact
request-path functions the parent doc names
(`groupMutations`→`observeMutation`, `observeRead` in
`kv/sharded_coordinator.go`), is allocation-free on the hot path, splits
reads vs writes, is windowed, has sub-range + Top-K split-key evidence,
and bounds cardinality. Driving detection off it avoids a second
parallel pipeline. The engine's dead
`RecordAccess`/`splitRange`/`Route.Load` path is removed in M3-PR1. M3
adds **zero** hot-path cost — it consumes already-flushed windows
off-path.
- **Aggregation — leader-local, no new RPC.** The detector runs on the
default-group leader and reads its own `MemSampler.Snapshot` for routes
it leads (the leader serves the load it would split on). No
`ReportAccess` RPC and no RPC piggybacking; cluster fan-out stays an
admin-UI concern and can slot behind the detector interface later if
needed.
- **Detector** — `score = write_ops*Ww + read_ops*Wr` (4/1), 50k ops/min
threshold, 3-consecutive-window up-side hysteresis + 0.8× down-side
band, per-route 10-min cooldown (monotonic deadline, not wall clock),
split-key from observed distribution (sub-range p50 / single-key
isolation / decline when no evidence — never blind midpoint),
guardrails: min route span, max routes, per-cycle split budget.
- **Scheduler** — invokes the existing `proto.Distribution.SplitRange`
RPC with `expected_catalog_version` CAS; **all catalog mutations go
through `SplitRange`** (repo convention). Same-group today; once M2
lands the **least-loaded-group** target policy flips the
`target_group_id` field on the same RPC.
- **Safety/ops** — default OFF behind `--autoSplit`; runtime kill switch
(`atomic.Bool`); fixed-enum metrics (no per-route/key labels); stable
slog keys; detector state is **leader-local and resets on election**
(re-earns confidence; cooldown re-seeded from `parent_route_id` lineage)
— stated explicitly so non-replication isn't mistaken for an oversight.
- **PR breakdown** — M3-PR1 (retire engine path + metric scaffolding),
M3-PR2 (aggregation + pure detector, observe-only), M3-PR3 (scheduler +
flag + kill switch), each independently shippable; M3-PR4 enables
cross-group target selection post-M2.

Every current-state claim cites concrete `file:line` evidence, verified
before writing.

## Open questions

1. **OQ-1** — Fully delete `Route.Load`/`Engine.Stats`, or keep `Stats`
for diagnostics?
2. **OQ-2** — Down-side hysteresis: hold the counter in the `[0.8×,
1.0×)` band, or decay it?
3. **OQ-3** — No sub-range/Top-K evidence: decline to split (proposed),
or allow a min/max-observed-key midpoint?
4. **OQ-4** — `minRouteSpan` without per-route key counts: approximate
from sampler evidence, or add a cheap key-count estimate?
5. **OQ-5** — Leader-local scoring sufficient, or is there a
heavy-follower-read workload needing cluster fan-out?
6. **OQ-6** — Post-M2: enable cross-group target automatically, or gate
behind `--autoSplitCrossGroup`?
7. **OQ-7** — Durable cooldown across elections: is `parent_route_id` +
a catalog split timestamp enough, or do we need durable `last_split_at`?
8. **OQ-8** — Express threshold in ops/min (proposed) or ops-per-`Step`
to avoid coupling sensitivity to `--keyvizStep`?

## Test plan

Doc-only — no Go tests run. Markdown verified clean; all cited
`file:line` references confirmed against `main`. The doc's own §8.2 lays
out the implementation-phase test strategy (table-driven detector unit
tests + `pgregory.net/rapid` invariants, detector→`SplitRange`
integration in the 3-node demo, Jepsen hotspot workload deferred to M4
per the parent doc).

Author: bootjp