Verify Slack failover alerts end to end in CI by reading the channel back through a Slack app

[anth-volk]

Follow-up to #1586 / #1587 (Slack failover lifecycle alerts, deployed in 0.27.0).

Problem

The alert path currently has no positive automated delivery verification. The webhook responds synchronously — HTTP 200 means Slack posted the message — but our _send only logs on failure (status ≥ 400 or exception), so "delivered" and "never attempted" are indistinguishable in logs. And even observing the 200 has one blind spot: it cannot detect a webhook pointing at the wrong channel, since Slack returns 200 regardless of which channel the webhook targets.

Each release run already produces a deterministic alert: the forced-fallback test against staging emits fallback_selected / forced_cloud_run for both channels through the deployed gateway. Today, confirming those messages actually rendered requires a human looking at the channel — which defeats the purpose once staging alerts move to a muted channel (see the planned environment-secret split for HOUSEHOLD_FAILOVER_SLACK_WEBHOOK_URL).

Proposal

Close the loop with a read-back assertion using a Slack app (free on all Slack plans):

1. Create an internal Slack app with a bot token carrying channels:history (plus channels:read if channel-ID lookup is needed), and invite the bot to the channel the staging webhook targets.
2. Store the bot token as a GitHub Actions secret (staging environment scope).
3. Add a CI step after the forced Cloud Run fallback test: call conversations.history on the channel and assert that alert messages exist for the current run's window — matching on the fields the alert payload carries (event fallback_selected, reason forced_cloud_run, environment staging, the Cloud Run revision, and a Created at timestamp inside the test window). Fail the job if no matching message is found.

This converts the per-release forced-fallback alert from incidental noise into an asserted end-to-end test of the entire chain: gateway event → notifier → webhook secret binding → Slack delivery → correct channel.

Notes

• Complementary smaller improvement (can be done independently or as part of this): log a structured success line in _send on 2xx so Cloud Logging holds a positive delivery receipt for every alert, including production ones.
• The 300s dedupe cooldown means the assertion should expect one message per channel (current/frontier) per run, not one per request.
• The assertion must tolerate the staging channel receiving alerts from concurrent runs (match on run window/revision, not just presence).
• If the staging webhook is later pointed at a Slack developer-sandbox workspace instead of a muted channel in the main workspace, the app/bot and token would live there instead — the mechanics are identical.