Check stability records (noise-damping stage 2) + the stages 2–4 plan

.workhorse/specs/monitoring/checks.md

@@ -76,6 +76,22 @@ All reported checks are kept, including passing ones, so that "every server repo
 A state whose effective result is warning or failed is an **issue**, eligible to contribute to incidents.
 "Degraded since" is the start of the current unbroken run of degradation; a recovery ends the run, and a later degradation starts a fresh one.
 
+### Stability
+
+Alongside each state, Canopy keeps a bounded stability record, updated as reports arrive.
+It is derived from **observed** results — untouched by policy, so operator grading and any noise damping built on the record never feed back into it.
+An observation is degraded when the observed result is warning, failed, or broken, and healthy when it is passed; skipped observations carry no signal and are not recorded.
+
+The record holds:
+
+- how many observations the state has received, and how many were degraded;
+- the most recent transitions between healthy and degraded, each with when it happened — a bounded ring, so a long-stable check remembers its distant history while a flapping one remembers only its recent churn;
+- an hour-of-week profile of how often the check is observed degraded, weighted towards recent weeks, so a load-dependent check (degraded during working hours, healthy overnight) is distinguishable from one degraded around the clock.
+
+From the ring, Canopy derives and presents how often the state has flapped recently and how long its degraded runs and healthy gaps typically last.
+The record is presented alongside the check on its check detail page and is available over the MCP interface (see [MCP](../private-server/mcp.md)).
+Nothing beyond the record is kept: it is a fixed-size summary per state, not a history.
+
 An effective broken result neither confirms nor clears the check's previous definite result: while broken, the state retains the contribution and degraded-since of its last definite effective result, and the brokenness itself additionally counts as a warning.
 A policy rule can grade brokenness differently — up to a failure where not being able to check is itself the failure, or down to a pass where a flaky check runner should not raise noise.
 A definite effective result (passed, warning, or failed) ends the broken condition and replaces the retained contribution.

.workhorse/specs/private-server/mcp.md

@@ -97,6 +97,9 @@ A summary or ranking of incidents should count published incidents rather than r
 **Get check documentation** takes a source and check name and returns the check's operator-authored markdown documentation (see [CHK](../monitoring/checks.md), "Documentation"), which by convention covers what the check observes, what each result means, and how to solve a failure.
 A client investigating an issue consults this before deriving a check's meaning from other sources.
 
+**Get check stability** takes a set of (source, check) pairs — optionally narrowed to one server or one group — and returns each matching state's full stability record (see [CHK](../monitoring/checks.md), "Stability"): the observation counts, the transition ring, and the hour-of-week degradation profile, together with the derived flap statistics (recent flap counts, typical degraded-run and healthy-gap durations).
+This is the raw material for analysing whether a check's noise is a flap, a load-dependent pattern, or a real change in behaviour.
+
 ## Result semantics
 
 A server's reported status reflects reports received within the recent-activity window; a server silent beyond that window reads as not recently seen rather than as a stale "up".

crates/canopy-mcp/src/incidents.rs

@@ -86,6 +86,53 @@ pub struct IssueIdArgs {
 	pub issue_id: String,
 }
 
+#[derive(Debug, Deserialize, JsonSchema)]
+pub struct CheckRefArg {
+	/// The source that reports the check (e.g. `alertd`, `canopy`).
+	pub source: String,
+	/// The check's name.
+	pub check_name: String,
+}
+
+#[derive(Debug, Deserialize, JsonSchema)]
+pub struct CheckStabilityArgs {
+	/// The (source, check) pairs to fetch stability for. Up to 32.
+	pub checks: Vec<CheckRefArg>,
+	/// Restrict to one server's id.
+	pub server_id: Option<String>,
+	/// Restrict to one group's id (its servers plus its group-scoped
+	/// checks).
+	pub group_id: Option<String>,
+}
+
+#[derive(Serialize)]
+struct CheckStabilityRow {
+	issue_id: Uuid,
+	/// The server the state belongs to; `null` for group- or canopy-wide
+	/// states.
+	server_id: Option<Uuid>,
+	server_name: Option<String>,
+	/// The group for group-scoped states; `null` otherwise.
+	group_id: Option<Uuid>,
+	source: String,
+	check_name: Option<String>,
+	observed_result: Option<CheckResult>,
+	effective_result: Option<CheckResult>,
+	active: bool,
+	/// The full stability record: observation counters, the
+	/// healthy↔degraded transition ring (oldest first), the hour-of-week
+	/// duty profile (168 buckets, UTC, Monday 00:00 first), and derived
+	/// flap statistics. `null` for states that predate stability
+	/// recording.
+	stability: Option<database::stability::StabilityData>,
+}
+
+#[derive(Serialize)]
+struct CheckStabilityOut {
+	/// One row per matching check state across all scopes.
+	rows: Vec<CheckStabilityRow>,
+}
+
 #[derive(Serialize)]
 struct IncidentSummary {
 	id: Uuid,
@@ -508,6 +555,72 @@ impl CanopyMcp {
 			documentation: policy.documentation,
 		})
 	}
+
+	#[tool(
+		description = "Full stability records for a set of checks, one row per (target, source, \
+		               check) state: observation counts, the recent healthy<->degraded transition \
+		               ring, an hour-of-week degradation profile (168 buckets, UTC, Monday 00:00 \
+		               first), and derived flap statistics (recent flip counts, typical \
+		               degraded-run and healthy-gap durations). Built from observed results, \
+		               before policy, so grading never distorts it. The raw material for telling \
+		               a flap from a load-dependent pattern from a real change in behaviour. \
+		               Optionally narrow to one server or one group."
+	)]
+	async fn get_check_stability(
+		&self,
+		Parameters(args): Parameters<CheckStabilityArgs>,
+	) -> Result<CallToolResult, McpError> {
+		if args.checks.is_empty() {
+			return Err(McpError::invalid_params(
+				"checks must name at least one (source, check_name) pair".to_string(),
+				None,
+			));
+		}
+		if args.checks.len() > 32 {
+			return Err(McpError::invalid_params(
+				"too many checks: at most 32 (source, check_name) pairs per call".to_string(),
+				None,
+			));
+		}
+		let server_id = parse_opt_uuid(&args.server_id, "server_id")?;
+		let group_id = parse_opt_uuid(&args.group_id, "group_id")?;
+		let pairs: Vec<(String, String)> = args
+			.checks
+			.into_iter()
+			.map(|c| (c.source, c.check_name))
+			.collect();
+
+		let mut conn = self.conn().await?;
+		let states = database::stability::states_for_checks(&mut conn, &pairs, server_id, group_id)
+			.await
+			.map_err(mcp_err)?;
+
+		let server_ids: Vec<Uuid> = unique(states.iter().filter_map(|(st, _)| st.server_id));
+		let names = Server::names_by_ids(&mut conn, &server_ids)
+			.await
+			.map_err(mcp_err)?;
+		let now = Timestamp::now();
+		let rows: Vec<CheckStabilityRow> = states
+			.into_iter()
+			.map(|(st, stability)| CheckStabilityRow {
+				issue_id: st.id,
+				server_id: st.server_id,
+				server_name: st
+					.server_id
+					.and_then(|sid| names.get(&sid))
+					.and_then(|(n, _)| n.clone()),
+				group_id: st.server_group_id,
+				source: st.source,
+				check_name: st.check_name,
+				observed_result: st.observed_result,
+				effective_result: st.effective_result,
+				active: st.active,
+				stability: stability
+					.map(|row| database::stability::StabilityData::from_row(&row, now)),
+			})
+			.collect();
+		ok_json(&CheckStabilityOut { rows })
+	}
 }
 
 /// How long the incident was (or has been) open, in seconds.

crates/database/src/issues.rs

@@ -298,7 +298,7 @@ async fn stamp_check_state(
 	} else {
 		prior_last_degraded.map(jiff_diesel::Timestamp::from)
 	};
-	diesel::update(issues::table.filter(issues::id.eq(issue_id)))
+	let issue = diesel::update(issues::table.filter(issues::id.eq(issue_id)))
 		.set((
 			issues::check_name.eq(&stamp.check),
 			issues::observed_result.eq(stamp.observed.to_string()),
@@ -311,7 +311,11 @@ async fn stamp_check_state(
 		.returning(Issue::as_select())
 		.get_result(conn)
 		.await
-		.map_err(AppError::from)
+		.map_err(AppError::from)?;
+	// Every stamped filing also feeds the state's stability record (from
+	// the observed result, so policy never feeds back into it).
+	crate::stability::record_observation(conn, issue_id, stamp.observed, at).await?;
+	Ok(issue)
 }
 
 impl NewEvent {
@@ -1042,11 +1046,11 @@ pub async fn health_from_check_state(
 }
 
 impl Issue {
-	/// Server-scoped check state for one (source, check). The per-check
-	/// attention page's data source: rows carry the observed/effective
-	/// results, the check's detail, and the degraded-streak timestamps.
-	/// A check's identity is the pair — a same-named check from another
-	/// source is a different check.
+	/// Check state for one (source, check), across every scope — server,
+	/// group, and canopy-wide. The check detail page's data source: rows
+	/// carry the observed/effective results, the check's detail, and the
+	/// degraded-streak timestamps. A check's identity is the pair — a
+	/// same-named check from another source is a different check.
 	pub async fn check_state_for_check(
 		conn: &mut AsyncPgConnection,
 		source: &str,
@@ -1058,7 +1062,6 @@ impl Issue {
 			.select(Issue::as_select())
 			.filter(dsl::source.eq(source))
 			.filter(dsl::check_name.eq(check_name))
-			.filter(dsl::server_id.is_not_null())
 			.filter(dsl::observed_result.is_not_null())
 			.load(conn)
 			.await

crates/database/src/lib.rs

@@ -28,6 +28,7 @@ pub mod servers;
 pub mod silenced_refs;
 pub mod slack_outbox;
 pub mod sql_playground_history;
+pub mod stability;
 pub mod statuses;
 pub mod tags;
 pub mod tailscale_users;

crates/database/src/schema.rs

@@ -185,6 +185,26 @@ diesel::table! {
 	}
 }
 
+diesel::table! {
+	check_stability (issue_id) {
+		issue_id -> Uuid,
+		created_at -> Timestamptz,
+		updated_at -> Timestamptz,
+		observations -> Int8,
+		degraded_observations -> Int8,
+		last_observed_at -> Nullable<Timestamptz>,
+		last_observed_degraded -> Nullable<Bool>,
+		transitions -> Jsonb,
+		duty_cycle -> Jsonb,
+	}
+}
+
+diesel::table! {
+	check_stability_backfill (done_at) {
+		done_at -> Timestamptz,
+	}
+}
+
 diesel::table! {
 	chrome_releases (version) {
 		version -> Text,
@@ -591,6 +611,7 @@ diesel::joinable!(device_connections -> devices (device_id));
 diesel::joinable!(device_keys -> devices (device_id));
 diesel::joinable!(device_server_associations -> devices (device_id));
 diesel::joinable!(device_server_associations -> servers (server_id));
+diesel::joinable!(check_stability -> issues (issue_id));
 diesel::joinable!(incident_issues -> incidents (incident_id));
 diesel::joinable!(incident_issues -> issues (issue_id));
 diesel::joinable!(incident_notes -> incidents (incident_id));
@@ -631,6 +652,8 @@ diesel::allow_tables_to_appear_in_same_query!(
 	backup_type_defaults,
 	bestool_snippets,
 	check_policies,
+	check_stability,
+	check_stability_backfill,
 	chrome_releases,
 	device_connections,
 	device_keys,