feat(observability): wire SNS notification action to DLQ CloudWatch alarms

[nizar-lahlali]

Background

Note: PR #208 (issue #117) is not merged yet — it is still open and under review. This is a follow-up to be picked up after #208 ships. The DLQ alarms described below do not exist on main today.

PR #208 / issue #117 adds CloudWatch DLQ-depth alarms (ApproximateNumberOfMessagesVisible >= 1, 5-min / Maximum, treatMissingData: NOT_BREACHING) for two stream consumers:

• FanOutConsumer — dlqDepthAlarm (cdk/src/constructs/fanout-consumer.ts)
• ApprovalMetricsPublisherConsumer — dlqAlarm (cdk/src/constructs/approval-metrics-publisher-consumer.ts)

Once merged, these alarms will ship without an addAlarmAction / SNS notification target — by design, as an intentional intermediate step (see docs/design/CEDAR_HITL_GATES.md §11.5). Alarm state is durable and queryable in the CloudWatch console, but there is no push notification when a DLQ starts accumulating poison-pill records. Operators must poll the Alarms console or configure a subscription manually.

Proposal

Provision an operational notification channel and wire it to the alarms introduced by #208:

1. Add an SNS topic (e.g. an OperationalAlerts construct) — ideally reusable across the stack, not per-consumer.
2. Attach it to both alarms via alarm.addAlarmAction(new cloudwatch_actions.SnsAction(topic)). Both alarm properties are exposed as cloudwatch.IAlarm specifically so a consumer can call addAlarmAction without changes to the constructs.
3. Support subscription target(s) — SNS → Slack / PagerDuty / email. Delivery target should be configurable (Blueprint prop / stack context) rather than hard-coded.

No metric or alarm restructuring is required — this is purely additive (per §11.5 "Follow-up — notification channel wiring").

Scope / considerations

• Wire the two DLQ-depth alarms above at minimum. Consider whether the GithubScreenshotIntegration DLQ alarm (mirrors the same threshold-1 shape) should share the topic.
• Once a channel exists, the "additional alarms (not yet shipped)" in §11.5 (e.g. high approval-timeout rate) become worth revisiting — but those can be separate follow-ups.
• Update docs/design/CEDAR_HITL_GATES.md §11.5 + the deferred-item note (~line 2111) once shipped, and regenerate the Starlight mirror (cd docs && node scripts/sync-starlight.mjs).

Dependencies

• Blocked by feat(cdk): add CloudWatch alarms for FanOut + ApprovalMetricsPublisher DLQs #208 — do not start until the DLQ alarms land on main.

References

• §11.5 "CloudWatch alarms" and the deferred-work note in docs/design/CEDAR_HITL_GATES.md
• Parent: Add CloudWatch alarms for FanOutConsumer + ApprovalMetricsPublisher DLQs #117 · PR: feat(cdk): add CloudWatch alarms for FanOut + ApprovalMetricsPublisher DLQs #208