Check stability records (noise-damping stage 2) + the stages 2–4 plan

[passcod]

🤖 Stacked on #371 (stage 1, the static linger). Implements stage 2 — transition aggregates of the noise-damping plan, plus the plan document itself (docs/plans/alerting-hysteresis.md, covering stages 2–4, the off-the-shelf survey, and the augurs maintenance check). No behaviour hooks into the incident pipeline yet — everything here is recording and read surfaces, immediately useful for analysis on its own.

The stability record (spec: CHK gains a "Stability" section)

Every stamped filing feeds a fixed-size check_stability row, 1:1 with the check state, built from observed results — pre-policy, so operator grading and any damping later built on these numbers can never feed back into them. Skipped observations carry no signal. The row holds:

• lifetime observation counters (total / degraded);
• a 32-entry healthy↔degraded transition ring — a stable check remembers far into the past, a flapper remembers only its recent churn, which is exactly the signal;
• a 168-bucket hour-of-week duty profile (UTC, Monday first) whose bucket counters halve at a cap (512 ≈ two months at minute cadence), so the profile leans towards recent weeks without unbounded growth.

Deliberately not a new append-only log (we just deleted events for being one): the record is a bounded summary per state, updated in place inside the existing filing transaction. Flap statistics — flips in 24h/7d, ring coverage, typical degraded-run and healthy-gap durations — are derived at read time, not stored.

Backfill — designed to not block ingestion, and marked for deletion

A one-shot pass on the monitor pod (not a data migration) replays the last 30 days of status history into the records, so duty profiles and rings are useful from day one. Why not a migration: migrations run concurrently with the live workload, and a single fleet-wide INSERT…SELECT would hold FK row locks (FOR KEY SHARE) on most live issues rows until commit — the filing path's SELECT … FOR UPDATE conflicts with those, so the whole replay would stall ingestion. Instead the pod replays one server per statement (riding the (server_id, created_at) statuses index; row locks held for milliseconds), gated by a completion-marker table plus an advisory lock so concurrent pods don't duplicate the scan; a crash mid-run resumes on the next startup, and rows that already exist are never overwritten. The table-creation migration carries a lock_timeout = 5s guard for the FK's momentary SHARE ROW EXCLUSIVE on issues, failing the migrate run fast instead of queueing ingestion writes behind it.

Everything transitional is tagged TODO(backfill-removal) (replay code, monitor-pod call, its test, marker table) — one grep finds the whole cleanup once every deployment has run it. Approximations vs live recording, noted in the code: only checks present in a push's health array are replayed (recovery-by-omission isn't), and canopy's own checks have no status history so they start cold.

The check detail page (renamed from "attention")

The per-check page at /healthchecks/:source/:check is now the check detail page — it was already the check's home (policy chips, documentation, now stability), and "attention" only describes one part of it. Three sections:

• About this check — the operator documentation (unchanged);
• Fleet stability (new) — every target's duty profile summed into one heatmap plus aggregate flap counts ("Across 12 servers, 1 group with a record: …"), so shared load-dependent patterns read at a glance even when no single target stands out;
• Needs attention — now in the standard list shape used across the UI: rank buckets in display order (unranked last), groups alphabetical within a bucket with the group name as a linked section heading, servers by kind then name, ungrouped trailing. Position no longer encodes urgency — the result chip does. Each row carries the Stability column and expands to the check's data plus the full stability record.

All scopes now show: check_state_for_check returns group- and canopy-scoped states too. A group's own state (backup health and the like) files under its group heading as a whole group row, bucketed by the group's status-page rank; canopy's self-monitoring state gets a trailing canopy section. Both feed the fleet rollup. Server rows gained rank/kind on the wire so the client reuses the shared ordering helpers.

The settings editor at /settings/healthchecks/:check is renamed HealthcheckSettings to free the name; the private endpoint follows (status_check_attention → status_check_detail, breaking only for the UI, which is its sole consumer).

MCP

get_check_stability: takes up to 32 (source, check) pairs, optionally narrowed to one server or group, and returns each matching state (all scopes) with its full record and derived stats — the raw material for agents to tell a flap from a load-dependent pattern from a real change.

Tests

check_stability suite: counters/ring/duty recording, skipped-carries-no-signal, observed-not-effective, ring bounding, bucket halving, backfill replay + marker gating + no-overwrite, pure derived-stats. MCP tool covered end-to-end. Playwright coverage for the stability column, per-server heatmap, no-record state, fleet rollup, and the group/canopy sections. Full database (167), private-server (242), statuses, and e2e (144) suites green.

Plan context

docs/plans/alerting-hysteresis.md tracks the staging: stage 1 ✅ #371, stage 2 ✅ this PR, stage 3 (adaptive linger + open grace from these aggregates) and stage 4 (flapping state, operator-confirmed pattern suggestions) still to come.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AEZZ9HMkKAFqrJzTg3FrE7