feat(warmer): herd mitigation for rolling deploys (#59) (#69)

* docs(spec): cache warmer herd mitigation design (#59)

A1 (jitter) + A2 (baseline delay) + A3 (paced batched warmup) + B2b
(N-slot PG advisory-lock semaphore with wait-on-miss) for the cluster-
wide inter-pod warmer stampede observed on rolling deploys.

Six new config knobs (cache-warm-delay/-jitter/-concurrency/-wait-max/
-batch-size/-batch-pause) with conservative defaults. Reuses the
session-level advisory lock pattern from startup_locks.py for slot
allocation. Last-pod latency analysis included.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(plan): implementation plan for #59 herd mitigation

Bite-sized TDD-ordered tasks (14 total) covering: kwarg extension,
A2+A1 delay/jitter, A3 batched warmup, B2b advisory-lock semaphore
helpers (try-once, retry-loop, release), warm() integration, storage
wiring, ZConfig keys, three DB integration tests, CHANGES entry, full
test + lint pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(warmer): add six herd-mitigation kwargs (off defaults) (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(warmer): A2+A1 — startup delay + jitter sleep (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(warmer): A3 — paced batched warmup (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(warmer): B2b — _try_acquire_slot_once helper (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(warmer): B2b — _acquire_slot with retry loop and wait cap (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(warmer): B2b — _release_slot helper (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(warmer): B2b — integrate slot lifecycle into warm() (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(storage): wire herd-mitigation knobs into CacheWarmer (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(config): six ZConfig keys for cache-warm herd mitigation (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(warmer): DB integration tests for B2b semaphore (#59)

- concurrency=1 serializes two warmers
- concurrency=2 allows two parallel, third waits
- crash-safe slot release on connection close

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: changelog entry for cache warmer herd mitigation (#59)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* style: ruff format + lint after herd-mitigation changes

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(warmer): code review fixes (#59)

- close lock connection on slot acquisition timeout (was leaking)
- continue scanning remaining slots when single-slot try_lock raises
  (was halting prematurely, deviating from spec)
- use fresh psycopg connection per warmer thread in DB integration
  tests (psycopg connections are not thread-safe)
- assert lock_conn.close() called on timeout path in unit test

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: jensens

CHANGES.md

@@ -1,5 +1,43 @@
 # Changelog
 
+## 1.13.0 (unreleased)
+
+### Features
+
+- **Cache warmer herd mitigation** (#59).  Rolling Kubernetes deploys
+  used to multiply DB primary CPU by the replica count for the warmer
+  startup window, because every pod independently fired its full set
+  of warmup `SELECT`s simultaneously.  Four composable behaviors fix
+  this, all wired through six new ZConfig keys:
+
+  - `cache-warm-delay` (default 15s): baseline sleep before warmer
+    starts. Moves the warmer out of the pod's own cold-start window
+    (Plone import, plone.pgcatalog schema check, ANALYZE chatter).
+  - `cache-warm-jitter` (default 30s): additional `random(0, jitter)`
+    sleep on top.  Spreads arrival times across pods.
+  - `cache-warm-concurrency` (default 2): maximum number of pods
+    warming in parallel, cluster-wide.  Enforced via a session-level
+    PostgreSQL advisory lock semaphore (same pattern as the existing
+    startup-DDL lock).  Pods that miss a slot retry with jittered
+    backoff until `cache-warm-wait-max` expires.
+  - `cache-warm-wait-max` (default 300s): retry cap before giving up
+    and skipping warmup (logged as WARNING).
+  - `cache-warm-batch-size` (default 500): zoids per `SELECT` batch.
+  - `cache-warm-batch-pause` (default 0.5s): sleep between batches.
+    Lowers per-pod peak qps.
+
+  **Observable behavior change:** warmer queries are now delayed by
+  ~15–45s post-startup (delay + jitter), not immediate.  To restore
+  pre-1.13 behavior, set `cache-warm-delay=0`, `cache-warm-jitter=0`,
+  `cache-warm-batch-pause=0`, `cache-warm-concurrency=9999`.
+
+  **Last-pod latency:** for a 6-pod fleet with default settings, the
+  last pod has a warm L2 cache approximately 45s after the deploy
+  window.  All pods serve traffic from T+0; this is the time-to-warm,
+  not the time-to-ready.
+
+  Design: [docs/superpowers/specs/2026-05-29-cache-warmer-herd-mitigation-design.md](docs/superpowers/specs/2026-05-29-cache-warmer-herd-mitigation-design.md).
+
 ## 1.12.0
 
 ### Features