docs(ci): publish CI alert runbook and validation evidence

[kami619]

Description

This PR completes RHAIENG-5938 by documenting and validating the existing shared-branch CI Slack alert flow without changing workflow behavior.
Changes in this PR:

• add docs/ci-alert-runbook.md as the evergreen responder runbook for CI alerts
• add docs/ci-alert-validation-report.md with live evidence from recent main failures and payload-quality verification
• update docs/ci-alert-policy.md and docs/ci-health-dashboard.md to link to the new runbook/evidence docs
  Motivation:
• make the current alert routing, severity, dedupe, and triage expectations explicit
• publish auditable validation evidence so the team can trust the current alerting behavior
• capture the current SLA measurement nuance for matrix workflows without changing runtime behavior

Jira Ticket

RHAIENG-5938

Testing

• make test passes (run from the affected agent directory)
• Manual testing performed (describe steps below)
• No testing required (documentation/config change only)
  Additional verification:
• uv run --no-sync python -m pytest .github/scripts/tests/test_notify_slack.py -v (12 passed)
• reviewed existing main GitHub Actions evidence for:
  • Inner Loop Gating run 29016905658
  • QG4: Agent Deployment Integration Tests run 29890165253
• rendered a representative Slack payload locally via .github/actions/notify-slack/render_payload.sh to confirm:
  • workflow name/header
  • event/ref fields
  • failed-job list
  • workflow-run and CI-dashboard links

Checklist

• I have read CONTRIBUTING.md
• No .env or secret files are included in this PR
• All changes are within scope of the linked Jira ticket (if not, explain in Description)

Review Guidance

Please review these files in order:

1. docs/ci-alert-runbook.md
2. docs/ci-alert-validation-report.md
   The changes in docs/ci-alert-policy.md and docs/ci-health-dashboard.md are just cross-links to the new docs.
   Important scope note:

• this PR is docs-only
• no GitHub Actions workflow behavior changed
• no Slack routing, gating, or payload logic changed

Related PRs

• feat(ci): add Slack alerts for shared-branch workflow failures #246
• docs(ci): define alert policy for QG signals #263

[coderabbitai]

Warning

Review limit reached

@kami619, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 58 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9d6ed313-e921-405f-990f-05f9fa8835d5

📥 Commits

Reviewing files that changed from the base of the PR and between 156a6ad and f9eefde.

📒 Files selected for processing (4)

• docs/ci-alert-policy.md
• docs/ci-alert-runbook.md
• docs/ci-alert-validation-report.md
• docs/ci-health-dashboard.md

📝 Walkthrough

Walkthrough

Changes

CI alert documentation

Layer / File(s)	Summary
Alert runbook and validation evidence docs/ci-alert-runbook.md, docs/ci-alert-validation-report.md	Adds runbook guidance for alert qualification, routing, deduplication, severity, SLAs, and responder steps, along with workflow validation evidence and payload verification details.
Policy and dashboard reference alignment docs/ci-alert-policy.md, docs/ci-health-dashboard.md	Updates timing language to reference workflow conclusion and links policy and dashboard guidance to the CI Alert Runbook.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: shricharan-ks

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: publishing the CI alert runbook and validation evidence.
Description check	✅ Passed	The description is directly related to the docs-only CI alert runbook and validation updates in this PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch rhaieng-5938-ci-alert-runbook

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mpk-droid]

Nit: The runbook (line 53) and policy doc (line 44) both say "within 5 minutes of workflow failure," but for matrix workflows the alert waits for all jobs to finish before sending. The QG4 validation sample shows 6m37s from first failure to notification, but only 6s from workflow conclusion.

Consider saying "within 5 minutes of workflow conclusion" so responders know what to expect for matrix workflows.

Assisted by Claude Opus 4.6 (1M context)

[mpk-droid]

lgtm. added nit