diff --git a/docs/ci-alert-policy.md b/docs/ci-alert-policy.md index 120d0abd..0aac4091 100644 --- a/docs/ci-alert-policy.md +++ b/docs/ci-alert-policy.md @@ -41,7 +41,7 @@ This MVP does not send team Slack alerts for: this MVP does not define an acknowledgement SLA, resolution SLA, or after-hours on-call expectation. - **System delivery expectation:** qualifying alerts should land in Slack within - 5 minutes of workflow failure. + 5 minutes of workflow conclusion. - **Severity model:** canonical severity fingerprinting is based on the Quality Gate ladder, with lower-numbered QGs treated as higher priority. - **Current implementation note:** not every current alert source is a canonical @@ -144,11 +144,10 @@ This MVP defines a system delivery path, not a human response SLA. 1. A qualifying shared-branch workflow run fails. 2. The repository posts one alert into the shared CI Slack destination. -3. The alert should arrive within 5 minutes of the qualifying failure. +3. The alert should arrive within 5 minutes of workflow conclusion. 4. The responder follows the workflow run link first, then uses the CI health dashboard for broader context. -5. Detailed triage steps live in the CI alert runbook that builds on this - policy. +5. Detailed triage steps live in the [CI Alert Runbook](./ci-alert-runbook.md). ### Human expectation diff --git a/docs/ci-alert-runbook.md b/docs/ci-alert-runbook.md new file mode 100644 index 00000000..5f0f2661 --- /dev/null +++ b/docs/ci-alert-runbook.md @@ -0,0 +1,84 @@ +# CI Alert Runbook + +## Scope + +This runbook covers the shared-branch CI Slack alerts for +`agentic-starter-kits`. + +| Workflow | Canonicality | Severity fingerprint | Qualifying events | +| --- | --- | --- | --- | +| `Code Quality` | Non-canonical supporting signal | None yet | `push` on `main`, `workflow_dispatch` on `main` | +| `Agent Tests` | Non-canonical supporting signal | None yet | `push` on `main`, `workflow_dispatch` on `main` | +| `Inner Loop Gating` | Canonical | `QG7` | `push` on `main` when matching eval/behavioral paths change, `workflow_dispatch` on `main` | +| `QG4: Agent Deployment Integration Tests` | Canonical | `QG4` | `schedule`, `workflow_dispatch` on `main` | + +## Routing and Ownership + +- **Route:** repository secret `SLACK_WEBHOOK_URL` +- **Accountable owner group:** `@aaet-tooling-experience` +- **Human handling model:** `notify-only` + +## Dedupe and Timing Semantics + +- One Slack alert is emitted per qualifying workflow run. +- The alert is emitted after the workflow conclusion is known. +- Matrix failures are aggregated into one workflow-level alert with a failed-job list. +- If the GitHub jobs API lookup fails, the alert still sends without job detail. + +## Severity Interpretation + +- `QG4` is the highest current action priority because lower-numbered canonical + QGs outrank higher-numbered ones. +- `QG7` is the next current canonical priority and should be triaged after + `QG4` but ahead of supporting signals. +- `Code Quality` and `Agent Tests` are supporting signals without canonical + fingerprints yet; treat them as operationally important but lower urgency + unless the failure is novel, repeated, or blocks shared-branch progress. + +## Non-alert Cases + +The current implementation does not send a team Slack alert for: + +- `pull_request` runs +- `workflow_dispatch` runs on non-`main` branches +- successful workflow runs + +If one of these runs fails, responders should expect no Slack message and +should inspect the GitHub Actions run directly. + +## SLA Path + +1. A qualifying shared-branch workflow run fails. +2. The repository posts one alert into the shared CI Slack destination. +3. The alert should arrive within 5 minutes of workflow conclusion. +4. The responder opens the workflow run first, then the CI dashboard. +5. The responder follows the validation and triage guidance in this runbook. + +## Responder Checklist + +1. Open the workflow run link from Slack and confirm the run is a qualifying + shared-branch failure. +2. Review the failed-job list in Slack; if it is missing, use the workflow run + page because the alert may have sent without job detail. +3. Use the workflow logs to decide whether the failure is transient, already + known, or a new defect. +4. Open the CI health dashboard to check whether the same workflow or related + shared-branch signals are already failing. +5. Prioritize follow-up using the severity interpretation in this runbook and + the proposed team SLA table below. +6. Route the follow-up through the normal team remediation or backlog path + because this MVP remains `notify-only`. + +## Proposed Team SLA (discussion item) + +| Severity bucket | Current signal mapping | Proposed acknowledgement window | Proposed triage target | +| --- | --- | --- | --- | +| Highest current priority | `QG4` | Within 1 business hour | Same business day | +| Next current priority | `QG7` | Within 4 business hours | Next business half-day | +| Supporting signals | `Code Quality`, `Agent Tests` | By next business day | Next business day or convert to backlog follow-up if duplicate | + +## Validation Evidence + +- Current evidence snapshot: [CI Alert Validation Report](./ci-alert-validation-report.md) +- The report records the exact `main` run IDs and observed notify-job timings + used to validate this runbook. diff --git a/docs/ci-alert-validation-report.md b/docs/ci-alert-validation-report.md new file mode 100644 index 00000000..2e168c5b --- /dev/null +++ b/docs/ci-alert-validation-report.md @@ -0,0 +1,50 @@ +# CI Alert Validation Report + +## Evidence Summary + +Timestamps below come from the GitHub Actions Jobs API (`gh run view --json jobs`) unless noted as a notify-step or log observation. + +| Workflow | Run ID | Event | Failure signal | Notify evidence | Observation | +| --- | --- | --- | --- | --- | --- | +| `Inner Loop Gating` | `29016905658` | `workflow_dispatch` on `main` | `Cluster Behavioral Tests`: failing step `Run shared btest runner` at `2026-07-09T12:07:04Z`; job `completedAt` `2026-07-09T12:07:08Z` | `Notify Slack` completed successfully; `Send Slack notification` step at `2026-07-09T12:07:38Z`; job `completedAt` `2026-07-09T12:07:41Z` | One workflow-level alert was sent 34 seconds after the failing step completed | +| `QG4: Agent Deployment Integration Tests` | `29890165253` | `schedule` on `main` | `langflow-simple-tool-calling-agent`: failing step `Run integration test` at `2026-07-22T04:06:44Z`; job `completedAt` `2026-07-22T04:06:46Z` | `Notify Slack` completed successfully; `Send Slack notification` step at `2026-07-22T04:13:21Z`; log contains `Slack notification sent` | Matrix failures are deduped into one alert after the workflow conclusion is known | + +## SLA Measurement Note + +- The current notify path emits one alert after the workflow conclusion is known, + not when the first matrix leg fails. +- For `QG4: Agent Deployment Integration Tests` run `29890165253`, the first + failing matrix step completed at `2026-07-22T04:06:44Z`, that job's Jobs API + `completedAt` was `2026-07-22T04:06:46Z`, the final non-notify job + `completedAt` was `2026-07-22T04:13:15Z`, and `Slack notification sent` was + logged at `2026-07-22T04:13:21Z`. +- Measured from workflow-level failure/conclusion readiness, the sample meets + the current expectation because the alert followed the matrix conclusion by + 6 seconds. +- Measured from the first failing matrix job, the same sample exceeds 5 minutes. + That measurement-basis ambiguity should be reviewed by the team as a follow-up + runbook/policy clarification. + +## Payload Quality Verification + +These observations come from the local `render_payload.sh` preview command in +`docs/ci-health-dashboard.md`, using representative `Code Quality` inputs. They +are not from a captured live `Code Quality` failure on `main`. + +- The rendered header preserves the workflow name: `CI Failure: Code Quality`. +- The payload fields preserve the expected event/ref context: + `workflow_dispatch` on `main`. +- The failed-job section lists the expected workflow-level job names: + `lint` and `type-check`. +- The payload includes direct links to both the workflow run and the CI + dashboard for responder navigation. + +## Coverage Review + +- `Code Quality` and `Agent Tests` both use the same `Notify Slack` job shape, + the same `should_notify.sh` gate, the same `.github/actions/notify-slack` + composite action, and the same dashboard URL wiring. +- Recent qualifying `main` failures were not available for `Agent Tests`, and + recent `Code Quality` failures were limited to PR history, so live evidence + for those two workflows is configuration-equivalence rather than a fresh + shared-branch failure sample. diff --git a/docs/ci-health-dashboard.md b/docs/ci-health-dashboard.md index 7b3634bf..a5ac0cf9 100644 --- a/docs/ci-health-dashboard.md +++ b/docs/ci-health-dashboard.md @@ -31,6 +31,9 @@ Pull request runs are excluded from the summary to keep the page focused on shar Routing, ownership, severity, and MVP handling expectations for these alerts are defined in [CI Alert Policy](./ci-alert-policy.md). +Responder workflow, dedupe semantics, and validation guidance live in +[CI Alert Runbook](./ci-alert-runbook.md). + ## Slack alerts The same four workflows also send immediate Slack alerts when a shared-branch run fails: