red-hat-data-services · kami619 · Jul 22, 2026 · Jul 22, 2026 · Jul 22, 2026 · Jul 22, 2026
@@ -41,7 +41,7 @@ This MVP does not send team Slack alerts for:
   this MVP does not define an acknowledgement SLA, resolution SLA, or after-hours
   on-call expectation.
 - **System delivery expectation:** qualifying alerts should land in Slack within
-  5 minutes of workflow failure.
+  5 minutes of workflow conclusion.
 - **Severity model:** canonical severity fingerprinting is based on the Quality
   Gate ladder, with lower-numbered QGs treated as higher priority.
 - **Current implementation note:** not every current alert source is a canonical
@@ -144,11 +144,10 @@ This MVP defines a system delivery path, not a human response SLA.
 
 1. A qualifying shared-branch workflow run fails.
 2. The repository posts one alert into the shared CI Slack destination.
-3. The alert should arrive within 5 minutes of the qualifying failure.
+3. The alert should arrive within 5 minutes of workflow conclusion.
 4. The responder follows the workflow run link first, then uses the CI health
    dashboard for broader context.
-5. Detailed triage steps live in the CI alert runbook that builds on this
-   policy.
+5. Detailed triage steps live in the [CI Alert Runbook](./ci-alert-runbook.md).
 
 ### Human expectation
 

@@ -0,0 +1,84 @@
+# CI Alert Runbook
+
+## Scope
+
+This runbook covers the shared-branch CI Slack alerts for
+`agentic-starter-kits`.
+
+| Workflow | Canonicality | Severity fingerprint | Qualifying events |
+| --- | --- | --- | --- |
+| `Code Quality` | Non-canonical supporting signal | None yet | `push` on `main`, `workflow_dispatch` on `main` |
+| `Agent Tests` | Non-canonical supporting signal | None yet | `push` on `main`, `workflow_dispatch` on `main` |
+| `Inner Loop Gating` | Canonical | `QG7` | `push` on `main` when matching eval/behavioral paths change, `workflow_dispatch` on `main` |
+| `QG4: Agent Deployment Integration Tests` | Canonical | `QG4` | `schedule`, `workflow_dispatch` on `main` |
+
+## Routing and Ownership
+
+- **Route:** repository secret `SLACK_WEBHOOK_URL`
+- **Accountable owner group:** `@aaet-tooling-experience`
+- **Human handling model:** `notify-only`
+
+## Dedupe and Timing Semantics
+
+- One Slack alert is emitted per qualifying workflow run.
+- The alert is emitted after the workflow conclusion is known.
+- Matrix failures are aggregated into one workflow-level alert with a failed-job list.
+- If the GitHub jobs API lookup fails, the alert still sends without job detail.
+
+## Severity Interpretation
+
+- `QG4` is the highest current action priority because lower-numbered canonical
+  QGs outrank higher-numbered ones.
+- `QG7` is the next current canonical priority and should be triaged after
+  `QG4` but ahead of supporting signals.
+- `Code Quality` and `Agent Tests` are supporting signals without canonical
+  fingerprints yet; treat them as operationally important but lower urgency
+  unless the failure is novel, repeated, or blocks shared-branch progress.
+
+## Non-alert Cases
+
+The current implementation does not send a team Slack alert for:
+
+- `pull_request` runs
+- `workflow_dispatch` runs on non-`main` branches
+- successful workflow runs
+
+If one of these runs fails, responders should expect no Slack message and
+should inspect the GitHub Actions run directly.
+
+## SLA Path
+
+1. A qualifying shared-branch workflow run fails.
+2. The repository posts one alert into the shared CI Slack destination.
+3. The alert should arrive within 5 minutes of workflow conclusion.
+4. The responder opens the workflow run first, then the CI dashboard.
+5. The responder follows the validation and triage guidance in this runbook.
+
+## Responder Checklist
+
+1. Open the workflow run link from Slack and confirm the run is a qualifying
+   shared-branch failure.
+2. Review the failed-job list in Slack; if it is missing, use the workflow run
+   page because the alert may have sent without job detail.
+3. Use the workflow logs to decide whether the failure is transient, already
+   known, or a new defect.
+4. Open the CI health dashboard to check whether the same workflow or related
+   shared-branch signals are already failing.
+5. Prioritize follow-up using the severity interpretation in this runbook and
+   the proposed team SLA table below.
+6. Route the follow-up through the normal team remediation or backlog path
+   because this MVP remains `notify-only`.
+
+## Proposed Team SLA (discussion item)
+
+| Severity bucket | Current signal mapping | Proposed acknowledgement window | Proposed triage target |
+| --- | --- | --- | --- |
+| Highest current priority | `QG4` | Within 1 business hour | Same business day |
+| Next current priority | `QG7` | Within 4 business hours | Next business half-day |
+| Supporting signals | `Code Quality`, `Agent Tests` | By next business day | Next business day or convert to backlog follow-up if duplicate |
+
+## Validation Evidence
+
+- Current evidence snapshot: [CI Alert Validation Report](./ci-alert-validation-report.md)
+- The report records the exact `main` run IDs and observed notify-job timings
+  used to validate this runbook.
@@ -0,0 +1,50 @@
+# CI Alert Validation Report
+
+## Evidence Summary
+
+Timestamps below come from the GitHub Actions Jobs API (`gh run view <run-id> --json jobs`) unless noted as a notify-step or log observation.
+
+| Workflow | Run ID | Event | Failure signal | Notify evidence | Observation |
+| --- | --- | --- | --- | --- | --- |
+| `Inner Loop Gating` | `29016905658` | `workflow_dispatch` on `main` | `Cluster Behavioral Tests`: failing step `Run shared btest runner` at `2026-07-09T12:07:04Z`; job `completedAt` `2026-07-09T12:07:08Z` | `Notify Slack` completed successfully; `Send Slack notification` step at `2026-07-09T12:07:38Z`; job `completedAt` `2026-07-09T12:07:41Z` | One workflow-level alert was sent 34 seconds after the failing step completed |
+| `QG4: Agent Deployment Integration Tests` | `29890165253` | `schedule` on `main` | `langflow-simple-tool-calling-agent`: failing step `Run integration test` at `2026-07-22T04:06:44Z`; job `completedAt` `2026-07-22T04:06:46Z` | `Notify Slack` completed successfully; `Send Slack notification` step at `2026-07-22T04:13:21Z`; log contains `Slack notification sent` | Matrix failures are deduped into one alert after the workflow conclusion is known |
+
+## SLA Measurement Note
+
+- The current notify path emits one alert after the workflow conclusion is known,
+  not when the first matrix leg fails.
+- For `QG4: Agent Deployment Integration Tests` run `29890165253`, the first
+  failing matrix step completed at `2026-07-22T04:06:44Z`, that job's Jobs API
+  `completedAt` was `2026-07-22T04:06:46Z`, the final non-notify job
+  `completedAt` was `2026-07-22T04:13:15Z`, and `Slack notification sent` was
+  logged at `2026-07-22T04:13:21Z`.
+- Measured from workflow-level failure/conclusion readiness, the sample meets
+  the current expectation because the alert followed the matrix conclusion by
+  6 seconds.
+- Measured from the first failing matrix job, the same sample exceeds 5 minutes.
+  That measurement-basis ambiguity should be reviewed by the team as a follow-up
+  runbook/policy clarification.
+
+## Payload Quality Verification
+
+These observations come from the local `render_payload.sh` preview command in
+`docs/ci-health-dashboard.md`, using representative `Code Quality` inputs. They
+are not from a captured live `Code Quality` failure on `main`.
+
+- The rendered header preserves the workflow name: `CI Failure: Code Quality`.
+- The payload fields preserve the expected event/ref context:
+  `workflow_dispatch` on `main`.
+- The failed-job section lists the expected workflow-level job names:
+  `lint` and `type-check`.
+- The payload includes direct links to both the workflow run and the CI
+  dashboard for responder navigation.
+
+## Coverage Review
+
+- `Code Quality` and `Agent Tests` both use the same `Notify Slack` job shape,
+  the same `should_notify.sh` gate, the same `.github/actions/notify-slack`
+  composite action, and the same dashboard URL wiring.
+- Recent qualifying `main` failures were not available for `Agent Tests`, and
+  recent `Code Quality` failures were limited to PR history, so live evidence
+  for those two workflows is configuration-equivalence rather than a fresh
+  shared-branch failure sample.
@@ -31,6 +31,9 @@ Pull request runs are excluded from the summary to keep the page focused on shar
 Routing, ownership, severity, and MVP handling expectations for these alerts are
 defined in [CI Alert Policy](./ci-alert-policy.md).
 
+Responder workflow, dedupe semantics, and validation guidance live in
+[CI Alert Runbook](./ci-alert-runbook.md).
+
 ## Slack alerts
 
 The same four workflows also send immediate Slack alerts when a shared-branch run fails: