Merge pull request #362 from beyondessential/incident-pipeline-specs

Monitoring pipeline rework: per-source checks, result-transform policies, check-state issues

Author: passcod

.workhorse/specs/monitoring/checks.md

@@ -0,0 +1,119 @@
+---
+id: CHK
+---
+
+# Check state
+
+Canopy's monitoring is organised around checks: named conditions, each with a current result, reported by sources or determined by Canopy itself.
+This spec covers the check-state model — targets, sources, results, policy, and the operator controls over them.
+How device reports arrive is the status contract (see [STA](../public-server/statuses.md)); how degraded checks aggregate into incidents is the incident spec (see [INC](incidents.md)).
+
+## Targets
+
+Every check is scoped to exactly one target: a server, a server group, or Canopy as a whole.
+
+Server checks come from sources reporting on that server, and from Canopy's own per-server determinations such as source staleness.
+Group checks are conditions Canopy determines about a group's control plane, such as backup maintenance health (see [BKJ](../jobs/backup.md)).
+Canopy-wide checks are Canopy monitoring its own operation (see [SELF](../private-server/self-alerts.md)).
+
+## Sources
+
+A source is a named reporter of checks, identified by a short string.
+Multiple sources may report on the same server, each concerned with part of the system, and each source's reports are independent: a report from one source says nothing about another source's checks.
+
+Two source names are reserved for Canopy itself: `canopy` for conditions Canopy determines on its own (staleness, reachability, backup health, key expiry, self-monitoring), and `manual` for conditions raised by operators.
+Reports arriving over the device API cannot use the reserved names.
+
+## Results
+
+A check's result is one of, in decreasing order of urgency:
+
+- **failed** — the condition is failing.
+- **warning** — the condition is degraded but not failing.
+- **broken** — the check itself could not run; the condition is unconfirmed either way.
+- **passed** — the condition holds.
+- **skipped** — the check deliberately did not run.
+
+Every check has two results: the **observed** result, what the source reported, and the **effective** result, what policy makes of it.
+The observed result is always recorded as reported; everything Canopy acts on — issues, incidents, health rollups — follows the effective result.
+
+## Policy
+
+Policy is a transformation of results: for each check it maps the observed result to the effective one.
+There is one vocabulary on both sides — policy speaks in results, and what a source is told about its checks is the policy itself, not a projection of it.
+
+Fleet-wide policy lives in a catalog keyed by (source, check).
+An entry carries:
+
+- a **ceiling** — the maximum effective result, on the urgency ordering: a ceiling of `failed` changes nothing, `warning` grades failures as warnings, `passed` means recorded but never alerting, and `skipped` additionally tells the source not to bother running the check.
+- optional **rules** — conditional transforms evaluated against the check's own detail, the report's server-wide detail, and the server's effective tags; a rule can move a result in any direction, including upward: a warning graded as a failure, or a pass with a particular detail graded as a warning.
+- an **escalates** flag — an effective failure of this check notifies immediately, bypassing incident grace (see [INC](incidents.md)).
+
+A check is registered in the catalog with a ceiling of `warning` the first time it is reported; operators adjust from there.
+Canopy's own checks register with the policy their condition warrants instead of the default.
+
+### Scoped policy
+
+Beyond the fleet catalog, a transform can be scoped to a target: per server, per group, or Canopy-wide.
+Transforms apply in order — fleet catalog, then group, then server — each acting on the previous effective result, so the most specific scope has the last word.
+
+The operator interface presents one scoped policy: the **silence**, a scoped ceiling of `skipped`, recording who silenced and when.
+A silenced check keeps recording its observed results; its effective result is skipped, so it raises nothing and counts nowhere.
+The model admits arbitrary scoped transforms; surfaces beyond the silence are deliberately not offered yet.
+
+## Documentation
+
+Each catalogued (source, check) can carry operator-authored documentation: a single markdown document.
+By convention it covers three things — a general description of what the check observes, what each result means (what makes it fail as opposed to warn), and hints for solving a failure — and the editor seeds new documentation with a template of those sections; Canopy attaches no meaning to the document's structure.
+Operators author and edit the documentation in the operator UI; it is presented alongside the check wherever its state is presented, and is available over the MCP interface (see [MCP](../private-server/mcp.md)) so agents work from curated knowledge about a check rather than deriving it.
+Canopy's own checks ship with their documentation.
+
+## State
+
+For each (target, source, check) Canopy keeps exactly one state: the observed and effective results, the detail the source attached to the check's most recent report, when the check was first and most recently reported, and — while it is degraded — when the current degradation began.
+All reported checks are kept, including passing ones, so that "every server reporting this check" is answerable without scanning history.
+
+A state whose effective result is warning or failed is an **issue**, eligible to contribute to incidents.
+"Degraded since" is the start of the current unbroken run of degradation; a recovery ends the run, and a later degradation starts a fresh one.
+
+An effective broken result neither confirms nor clears the check's previous definite result: while broken, the state retains the contribution and degraded-since of its last definite effective result, and the brokenness itself additionally counts as a warning.
+A policy rule can grade brokenness differently — up to a failure where not being able to check is itself the failure, or down to a pass where a flaky check runner should not raise noise.
+A definite effective result (passed, warning, or failed) ends the broken condition and replaces the retained contribution.
+
+## Reporting semantics
+
+A source's report for a server carries that source's complete current set of checks.
+Canopy trusts the reporter: a check the source previously reported but omits from its current report has recovered, and its state records that.
+Omission by one source never affects another source's checks.
+
+## Source staleness
+
+A source that has reported on a server is expected to keep reporting.
+When a source's most recent report for a server is older than the server's down threshold, Canopy raises a staleness check for that (server, source) under the `canopy` source, and clears it when the source reports again.
+A server all of whose sources are stale is presented as unreachable.
+
+## Health rollup
+
+A server's health is derived from its current effective results across all sources: any failure makes it unhealthy; otherwise any warning or brokenness makes it degraded; otherwise it is healthy.
+Passed and skipped checks do not count against a server.
+
+## Operator controls
+
+**Silences** are the scoped policy described above.
+
+**Snoozes** suppress one state until a chosen time, after which it contributes again if still degraded.
+
+**Resolution** is an operator marking one state as dealt with, recording who and why.
+A resolved state that degrades again reopens: the resolution is cleared and the state contributes anew.
+
+**Notes** attach free-form operator commentary to a state.
+
+## Manual conditions
+
+Operators can raise a condition directly against a server, under the `manual` source, with a chosen check name, result, and message, and optionally marked as escalating.
+A manual condition behaves as a reported check whose reporter is the operator: it stays active until an operator resolves it or raises it again as recovered.
+
+## Monitoring gate
+
+Server-targeted checks on a server that is not monitored are recorded and presented for visibility but do not contribute to incidents.
+Group and Canopy-wide checks are not subject to any server's monitoring gate.

.workhorse/specs/monitoring/incidents.md

@@ -0,0 +1,43 @@
+---
+id: INC
+---
+
+# Incidents
+
+An incident is a span of trouble on a target: a server group, or Canopy as a whole.
+It aggregates the issues active on that target over its lifetime, from when it opens until it closes or an operator resolves it.
+At most one incident is open per target at a time.
+
+Issues and effective results are defined by the check-state model (see [CHK](checks.md)).
+Issues on a server belong to the target of the server's group; issues on an ungrouped server belong to no target and cannot contribute to incidents.
+Group-targeted issues belong to that group's target; Canopy-wide issues belong to the Canopy target.
+
+## Membership
+
+An incident opens when a check's effective result becomes failed on a target with no open incident.
+While an incident is open, every issue on its target joins it — effective warnings included — so the incident carries the full context of what was wrong during its span.
+
+An issue leaves the incident when it stops being one: its effective result recovers (to passed or skipped, whether by report or by policy), it is resolved or snoozed, or its server stops being monitored.
+The incident closes when its last effective failure leaves; warnings never hold an incident open.
+
+The membership history — which issues joined and left, and when — is kept and presented as the incident's timeline.
+An issue can leave and rejoin the same incident.
+
+Operator actions that change what counts (monitoring toggles, group membership changes, policy and silence changes) re-evaluate the affected issues' incident membership.
+
+## Notification
+
+Operators are notified over the notification channel: group incidents to the group's configured channel, Canopy-wide incidents to the operator channel.
+
+An incident notifies when it has stayed open past its target's grace period; an incident that closes within grace never notifies.
+Whether an incident notified is recorded as its **published** flag, so flaps can be excluded from reporting.
+An escalating check's effective failure (see [CHK](checks.md), "Policy") notifies immediately, bypassing any remaining grace; if the incident has already notified, the join escalates it with a further notification, at most once per incident.
+A notified incident notifies again when it closes.
+
+## Resolution
+
+An operator can resolve an open incident, recording who and why.
+Resolution cascades to the incident's open issues — each is resolved with the same attribution — and the incident closes as its members leave.
+Unresolving clears the resolution record; it does not reopen the incident.
+
+Notes attach free-form operator commentary to an incident.

.workhorse/specs/private-server/mcp.md

@@ -74,26 +74,28 @@ When the result is truncated to its bound, the result says so, so the client doe
 
 **Fleet summary** takes no input and returns a fleet-wide overview: server counts by kind and rank, the distribution of deployed versions, a rollup of server health, the number of groups, and a rollup of backup health.
 
-**Find backup problems** optionally narrows to one group, otherwise scans the whole fleet, and returns the current backup problems with a severity for each: server-and-type pairs whose last successful backup is overdue against its schedule, types that have never reported a backup, groups whose backup repository is in an error state, recent failed backup runs, and maintenance runs that appear stuck.
+**Find backup problems** optionally narrows to one group, otherwise scans the whole fleet, and returns the current backup problems graded by urgency: server-and-type pairs whose last successful backup is overdue against its schedule, types that have never reported a backup, groups whose backup repository is in an error state, recent failed backup runs, and maintenance runs that appear stuck.
 
 ### Incidents and issues
 
-An issue is a per-server (or per-group) condition raised from a known source under a stable reference, carrying a severity, a current active state, and a history of events; an incident aggregates the issues active for a group over a span of time, from when it opened until it closes or an operator resolves it.
+An issue is a degraded check state (see [CHK](../monitoring/checks.md)): a condition on a server, a group, or Canopy itself, reported by a known source under a stable check name, carrying its observed and effective results and a current active state; an incident aggregates the issues active on a target over a span of time, from when it opened until it closes or an operator resolves it (see [INC](../monitoring/incidents.md)).
 
 **Find incidents** takes a look-back window (in days, defaulting to a week) and optionally one group, and returns the incidents that were open at any point within that window — those still open, plus those that closed no earlier than the window start.
-Each is returned with its group, its status (open, closed, or operator-resolved), when it opened, closed, and was resolved, who resolved it and why, whether it ever escalated, how long it was open, whether it was published, and how many issues and events it covers.
+Each is returned with its target, its status (open, closed, or operator-resolved), when it opened, closed, and was resolved, who resolved it and why, whether it ever escalated, how long it was open, whether it was published, and how many issues it covers.
 A status filter can narrow the result to only open or only resolved incidents.
 
 The window includes incidents that flapped open and shut within their group's grace period and so never surfaced to anyone.
-Each incident therefore carries a **published** flag — true when it actually notified operators, which happens only when it stayed open past its group's grace period or it escalated (a critical issue joined, which bypasses the grace) — and the result reports how many of the returned incidents were published.
-The raw event count an incident accumulated is not a measure of its duration or severity: a high count can belong to a sub-minute flap.
+Each incident therefore carries a **published** flag — true when it actually notified operators, which happens only when it stayed open past its target's grace period or it escalated (an escalating check's failure joined, which bypasses the grace) — and the result reports how many of the returned incidents were published.
 A summary or ranking of incidents should count published incidents rather than raw rows unless raw activity is explicitly wanted.
 
-**Get incident** takes an incident identifier and returns the incident with the issues attached to it: each issue's severity, source, reference, message, owning server, active state, and when it joined and (if applicable) left the incident.
+**Get incident** takes an incident identifier and returns the incident with the issues attached to it: each issue's effective result, source, check name, message, owning server, active state, and when it joined and (if applicable) left the incident.
 
-**Find issues** returns issues across the fleet, filtered by active state, by severity, by group, by server, and by recency (issues last seen within a look-back window).
+**Find issues** returns issues across the fleet, filtered by active state, by effective result, by group, by server, and by recency (issues last seen within a look-back window).
 
-**Get issue** takes an issue identifier and returns the issue with its recent events and the incidents it is or was part of.
+**Get issue** takes an issue identifier and returns the issue with the incidents it is or was part of.
+
+**Get check documentation** takes a source and check name and returns the check's operator-authored markdown documentation (see [CHK](../monitoring/checks.md), "Documentation"), which by convention covers what the check observes, what each result means, and how to solve a failure.
+A client investigating an issue consults this before deriving a check's meaning from other sources.
 
 ## Result semantics
 

.workhorse/specs/private-server/self-alerts.md

@@ -5,28 +5,26 @@ id: SELF
 # Self-alerts
 
 A self-alert is Canopy reporting a problem with its own operation, as distinct from an issue observed on a fleet member.
+Self-alerts are Canopy-wide checks (see [CHK](../monitoring/checks.md)): they carry the same state, policy, silences, and resolution as any other check, and they aggregate into incidents on the Canopy target (see [INC](../monitoring/incidents.md)).
 
 ## Conditions
 
-Each self-alert condition is identified by a stable reference, and at most one alert exists per condition: repeated detections coalesce into the one alert rather than accumulating.
-An alert is active while its condition holds, and recovers when the condition clears; a condition without automatic recovery stays active until an operator resolves it.
+Each self-alert condition is a check with a stable name under the `canopy` source, and at most one state exists per condition: repeated detections update the one state rather than accumulating.
+A condition is active while it holds, and recovers when it clears; a condition without automatic recovery stays active until an operator resolves it.
 
 The current conditions are:
 
-- Canopy's own identity for reaching backup storage is broken (critical).
+- Canopy's own identity for reaching backup storage is broken (escalating).
 - An operator-notification delivery has permanently failed (stays until operator-resolved).
 - An MCP access token is within fifteen days of its expiry (see [MCP](mcp.md), "Access tokens").
 
 ## Notification
 
-An active self-alert notifies operators over the operator-notification channel directly: it does not open an incident and does not attach to any fleet group.
-One notification is sent when an alert becomes active, and one when it recovers.
-Alerts below critical severity wait out a short grace period before notifying, and an alert that recovers within that grace sends nothing at all; critical alerts notify immediately.
-While an alert stays active, repeated detections send no further notifications.
-When the notification channel is not configured for self-alerts, they are still recorded and presented, and the operator is warned that notification is off.
+Self-alerts notify through the incident machinery: an effective failure opens an incident on the Canopy target, which notifies the operator channel per the incident notification rules (grace period, escalation, recovery notice).
+When the notification channel is not configured, self-alerts are still recorded and presented, and the operator is warned that notification is off.
 
 ## Presentation
 
 Active self-alerts are presented on their own surface in the operator UI, apart from fleet issues and incidents: a persistent notice visible from any page while any alert is active, leading to a view of the active alerts and recent recoveries.
 Self-alerts do not appear in the fleet issue listings, and are not presented as belonging to any server.
-Each alert presents its severity, when it became active, and a description of the condition and what to do about it.
+Each alert presents its effective result, when it became active, and a description of the condition and what to do about it.