observability(ci): NR CI-health dashboard + alerts + synthetic money-journey alert + runbook (Wave 5)#57
Merged
Conversation
…journey alert + runbook (Wave 5) Wave 5 of the CI integration plan (docs/ci/01-CI-INTEGRATION-DESIGN.md §NR observability): make any CI / test / deploy failure studyable from New Relic, and add the alerts/tiles for the worker's new money/value synthetic journeys (rule 25 — every new metric/leg ships its alert + tile + catalog row). CI health (reads InstantCITestRun / InstantCITestFailure custom events the .github/actions/nr-ci-event composite action POSTs from api/worker/web CI): - newrelic/dashboards/instanode-ci-health.json — pass rate by repo, suite duration p50/p95 trend, failure count by workflow/suite, flake signal, live-prod e2e + pr-smoke results, recent-failures table (failed_step+log_url), deploy-gate results, run-volume (confirms emit wired). - newrelic/alerts/ci-failure-rate-spike.json (P2 — CI failure rate >40%/1h), ci-e2e-prod-failing.json (P1 — user journeys broke in prod), ci-pr-smoke-failing-main.json (P1 — UI↔API contract broke post-merge on main). Synthetic money/value journeys (worker flow_synthetic_money.go, separate PR): - newrelic/alerts/flow-test-money-fail.json (P1) — claim / deploy_status / checkout / magic_link fail, FACET flow; distinct P1 from the P0 read-path alert. The existing flow matrix dashboard + flow-test-p0-fail already FACET by flow, so the new flows surface automatically. Docs: - observability/METRICS-CATALOG.md — money-flow row update + a new "CI custom events" section documenting InstantCITestRun/Failure (NR Event API, not Prom). - FLOW-SYNTHETIC-RUNBOOK.md — the operator enable runbook for continuous prod synthetic monitoring (FLOW_SYNTHETIC_ENABLED + AUTH_PROBE_BEARER_TOKEN), the flows + truth surfaces, the alerts that light up, and the instant kill switch. All JSON validates (jq) + apply.sh --dry-run. infra has NO auto-apply (rule 15); operator applies via newrelic/apply.sh with NEW_RELIC_API_KEY + ACCOUNT_ID. Pre-existing repo debt unrelated to this change: a git merge-conflict marker in newrelic/tests/apply.test.sh (committed at b41339a) and 2 stale slo.test.sh livenessProbe assertions — not touched here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wave 5 of the CI integration plan (
docs/ci/01-CI-INTEGRATION-DESIGN.md§NR observability). Makes any CI / test / deploy failure studyable from New Relic, and ships the alerts/tiles for the worker's new money/value synthetic journeys (rule 25).CI health (NR custom events)
Reads
InstantCITestRun/InstantCITestFailureevents the.github/actions/nr-ci-eventcomposite action POSTs from api/worker/web CI (companion PRs below).newrelic/dashboards/instanode-ci-health.json— pass rate by repo, suite duration p50/p95 trend, failure count by workflow/suite, flake signal, live-prod e2e + pr-smoke results, recent-failures table (failed_step + log_url), deploy-gate results, run-volume.newrelic/alerts/ci-failure-rate-spike.json(P2 — CI failure rate >40%/1h)newrelic/alerts/ci-e2e-prod-failing.json(P1 — user journeys broke in prod)newrelic/alerts/ci-pr-smoke-failing-main.json(P1 — UI↔API contract broke post-merge on main)Synthetic money/value journeys (worker
flow_synthetic_money.go)newrelic/alerts/flow-test-money-fail.json(P1) — claim / deploy_status / checkout / magic_link fail, FACET flow. The existing flow matrix dashboard +flow-test-p0-failalready FACET by flow, so the new flows surface automatically.Docs
observability/METRICS-CATALOG.md— money-flow row + new "CI custom events" section.FLOW-SYNTHETIC-RUNBOOK.md— operator enable runbook (FLOW_SYNTHETIC_ENABLED+AUTH_PROBE_BEARER_TOKEN), flows + truth surfaces, alerts, kill switch.Verify
jq) +newrelic/apply.sh --dry-runprocesses every new file with no errors.newrelic/apply.shwithNEW_RELIC_API_KEY+NEW_RELIC_ACCOUNT_ID.Pre-existing repo debt (NOT touched here, flagged)
newrelic/tests/apply.test.sh(committed at b41339a) and 2 staleslo.test.shlivenessProbe assertions. These are not part of thevalidate.ymlCI gate.Companion PRs (Wave 5)
🤖 Generated with Claude Code