observability: flow-synthetic matrix dashboard + alerts + Prom rules + catalog

[mastermanas805]

What

The infra half of the continuous-monitoring synthetic flow runner — paired with worker PR feat/flow-synthetic-runner (#91). Turns the runner's three metrics + the InstantFlowTest NR event into the green/red matrix the directive asked for, with full rule-25 coverage (alert + dashboard tile + catalog row, all in one PR).

The two PRs ship together; rule 25's "same PR" is satisfied cross-repo.

Dashboard — newrelic/dashboards/instanode-reliability.json (5 new tiles)

• Flow matrix grid — SELECT latest(result) FROM InstantFlowTest WHERE cohort='synthetic' FACET flow, actor rendered as a table, red on fail. One cell per flow×actor — the single "is everything green right now" view.
• Fails by flow (1h, must be 0) and leaked reaps (1h, must be 0).
• P95 latency per flow (6h) — latency-regression watch.
• Distinct flows reporting (15m) — silent-death watch (a flow that stops emitting is itself a failure).

Alerts — newrelic/alerts/

file	condition	priority
flow-test-p0-fail.json	instant_flow_test_total{result="fail"} ABOVE 0 over 10m, FACET flow	P0
flow-test-latency-regression.json	percentile(latencyMs,95) > 5s over 30m	P2
flow-test-silent-death.json	event count BELOW 1 over 15m (loss-of-signal)	P1
flow-synthetic-leak.json	reaped{outcome="leaked"} ABOVE 0 (rule-24 cleanup-ledger breach)	P2

The P0 alert FACETs by flow; the description instructs the responder to read the failing event's commitId and compare to /healthz .commit_id to name the bad deploy (rule 14/15).

Prom backstop — k8s/prometheus-rules.yaml

New group instant-worker-flow-synthetic mirroring all four NR alerts (FlowTestP0Fail / FlowTestLatencyRegression / FlowTestSilentDeath / FlowSyntheticLeak). yamllint + (CRD-aware) kubeconform clean.

Catalog — observability/METRICS-CATALOG.md

Rows for instant_flow_test_total, instant_flow_test_latency_seconds, instant_flow_synthetic_reaped_total (lazy *Vec, INERT until the flag is on).

Inert until lit + operator steps (infra has NO auto-apply — CLAUDE.md rule 15)

All metrics are lazy *Vec and inert until an operator sets FLOW_SYNTHETIC_ENABLED=true on the worker — the alerts never fire until the flag is lit and the series exist. After both PRs merge + deploy, the operator must:

1. Light the worker flag: kubectl set env -n instant-infra deploy/instant-worker FLOW_SYNTHETIC_ENABLED=true (the worker PR's operator steps).
2. Apply the Prom rules: kubectl apply -f k8s/prometheus-rules.yaml (manual by design).
3. Import the NR dashboard (instanode-reliability.json) + create the alert policies from newrelic/alerts/flow-*.json, and wire them to the existing PagerDuty/Slack notification channel (same routing as auth-probe-fail.json).
4. NR license key for the metrics path is already present; the InstantFlowTest event push needs NR configured on the worker (else the runner degrades to a noop emitter — metrics still flow).

🤖 Generated with Claude Code