obs(wave-2): NR alerts + Prom rules + reliability dashboard for 12 new metrics

Wave 2 observability sweep — every new metric shipped between 2026-05-15 and
2026-05-20 now has a P0/P1/P2 alert AND a dashboard tile AND a row in
observability/METRICS-CATALOG.md. Per project memory `feedback-nr-observability-per-change`
and CLAUDE.md Rule 25 ("Every new metric ships with its alert + dashboard tile
in the same PR").

NR alerts added (newrelic/alerts/*.json — 11 files):
  - propagation-dead-lettered.json          → P0 (paid-customer regrade fell through)
  - propagation-unknown-kind.json           → P1 (api/worker image skew)
  - propagation-unexpected-skip.json        → P0 (CHAOS F1 sentinel)
  - orphan-sweep-no-db-row.json             → P0 (P0-3 atomic-provision symptom)
  - orphan-sweep-stuck-build-spike.json     → P1 (build pipeline degraded)
  - orphan-sweep-reap-failed.json           → P1 (reap path itself broken)
  - brevo-send-errors-spike.json            → P1 (SEND-side email pipeline)
  - magic-link-email-rate-limited.json      → P2 (abuse / NAT throttle)
  - provisioner-circuit-open.json           → P1 (backend down, fail-fast active)
  - email-missing-renderer.json             → P0 (silent customer-facing breakage)
  - worker-migration-mismatch.json          → P1 (api/worker schema disagreement)

Prometheus rules mirrored into k8s/prometheus-rules.yaml (8 new rule groups):
  - instant-worker-propagation-skip         (PropagationUnexpectedSkip)
  - instant-worker-brevo-send               (BrevoSendErrorsSpike + Warning)
  - instant-api-magic-link                  (MagicLinkEmailRateLimited)
  - instant-provisioner-circuit             (ProvisionerCircuitOpen + HalfOpen)
  - instant-worker-email-renderer           (EmailMissingRenderer)
  - instant-readyz                          (ReadyzCheckFailed + Degraded — mirror of prometheus/alert-rules.yml)
  - instant-api-brevo-webhook               (BrevoDeliveryRatioLow + Warn — mirror of NR email-delivery-ratio)
  - instant-worker-billing                  (BillingChargeUndeliverable)

Dashboard (newrelic/dashboards/instanode-reliability.json):
  15 tiles spanning /readyz across 3 services, propagation queue depth +
  dead-letter rate + per-kind retry distribution, orphan sweep activity,
  magic-link rate-limit, Brevo send/error/webhook funnel, provisioner
  circuit-breaker state, missing-renderer events, storage presign rate,
  billing charge undeliverable, worker migration drift.

METRICS-CATALOG (observability/METRICS-CATALOG.md):
  New doc enumerating every metric, its labels, emit timing (eager vs
  lazy *Vec), NR alert path, Prom rule name, and dashboard tile title.
  Lazy-emit gotcha documented so operators don't panic when a fresh
  deploy's /metrics page lacks an instant_*_total{label=...} series until
  the codepath fires.

CLAUDE.md Rule 25 strengthened:
  Was "alert OR dashboard"; now requires BOTH plus METRICS-CATALOG row
  in the same PR. The Wave 2 sweep found 14 metrics with zero monitoring
  — that's the failure mode this rule prevents.

DEPLOY:
  No auto-apply for infra repo. Operator applies via
  `infra/newrelic/apply.sh` (NerdGraph create/update mutations) and
  `kubectl apply -f infra/k8s/prometheus-rules.yaml`. The shell apply is
  idempotent — delete-then-create per existing alert pattern.

COVERAGE BLOCK (per CLAUDE.md rule 17):
  Symptom:        14 metrics shipped 2026-05-15..2026-05-20 with no NR
                  alert / no Prom rule / no dashboard tile.
  Enumeration:    `grep -nE 'instant_.*_total|brevo_.*_total|readyz_check_status|circuit_state|email_missing_renderer' api/internal/metrics/*.go worker/internal/metrics/*.go provisioner/internal/circuit/*.go`
  Sites found:    13 unique Prometheus metrics + 1 healthz log field
  Sites touched:  14 (all covered) — see METRICS-CATALOG.md table
  Coverage test:  jq empty over newrelic/alerts/*.json + python3 yaml-safe-load
                  over k8s/prometheus-rules.yaml + jq empty over
                  newrelic/dashboards/instanode-reliability.json — all PASS
  Live verified:  N/A for declarative configs — verification is
                  `bash newrelic/apply.sh` (operator) + observing the new
                  alert NRQL evaluating against live Metric stream within
                  ~2 min of apply.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: mastermanas805

k8s/prometheus-rules.yaml

@@ -279,3 +279,259 @@ spec:
               clean it — a k8s API outage or a DB write failure. Single transient events
               are fine; a sustained rate means the reap path itself is broken. Check
               instant-worker pod logs for `jobs.orphan_sweep.*_delete_failed` lines.
+
+    # CHAOS F1 (2026-05-20) — propagation_runner used to silently mark APPLIED on
+    # any row whose target resource was missing/in an unexpected state. The fix
+    # added unexpected_skip counting AND treats those rows as Failure (counts
+    # against maxAttempts). This rule pages if the counter ticks more than once
+    # — every unexpected_skip is one class of customer-facing nondeterminism
+    # that the operator should investigate, not just the eventual dead-letter.
+    - name: instant-worker-propagation-skip
+      rules:
+        - alert: PropagationUnexpectedSkip
+          expr: |
+            sum(rate(instant_propagation_unexpected_skip_total[15m])) by (kind, resource_type, skip_reason) > 0
+          for: 15m
+          labels:
+            severity: critical
+            service: worker
+          annotations:
+            summary: "propagation_runner unexpected_skip {{ $labels.kind }} / {{ $labels.resource_type }} / {{ $labels.skip_reason }}"
+            description: |
+              instant_propagation_unexpected_skip_total{kind="{{ $labels.kind }}",resource_type="{{ $labels.resource_type }}",skip_reason="{{ $labels.skip_reason }}"} > 0 for >15m.
+              The post-CHAOS-F1 fix means the row is now counted as a Failure (will eventually dead-letter and fire PropagationDeadLettered above) — but the operator should investigate the schema/state drift NOW. Find the row in pending_propagations, walk back to the api caller, and decide whether the missing resource should be inserted, the propagation row should be aborted, or the underlying race condition should be patched.
+
+    # Email pipeline SEND-side (worker → Brevo /v3/smtp/email). RECEIVE-side
+    # (Brevo → /webhooks/brevo) is covered by the NR email-delivery-ratio
+    # alert. The classification label separates fixable-by-key-rotation
+    # (permanent: 401/403) from fixable-by-backoff (transient: 5xx, 429) so the
+    # operator knows which runbook to follow.
+    - name: instant-worker-brevo-send
+      rules:
+        - alert: BrevoSendErrorsSpike
+          expr: |
+            sum(rate(brevo_send_errors_total[10m])) by (classification, status_code) * 600 > 10
+          for: 10m
+          labels:
+            severity: critical
+            service: worker
+          annotations:
+            summary: "brevo_send_errors_total > 10 in 10m (classification={{ $labels.classification }} status={{ $labels.status_code }})"
+            description: |
+              Worker → Brevo /v3/smtp/email is failing at >10 errors / 10m. Brevo
+              returns 201 the instant it accepts a POST, so a degraded send path
+              never affects the delivery-ratio alert — the messages simply never
+              queue at Brevo. Check classification: `permanent` (401/403) means
+              rotate BREVO_API_KEY; `transient` (5xx/429) means upstream
+              brown-out, the worker's exponential backoff will catch up.
+
+        - alert: BrevoSendErrorsWarning
+          expr: |
+            sum(rate(brevo_send_errors_total[10m])) by (classification, status_code) * 600 > 3
+          for: 10m
+          labels:
+            severity: warning
+            service: worker
+          annotations:
+            summary: "brevo_send_errors_total > 3 in 10m (early warning) classification={{ $labels.classification }}"
+            description: |
+              Lower-severity early-warning twin of BrevoSendErrorsSpike. Lets the
+              operator catch a slow upstream brown-out before it crosses the page
+              threshold. No paging — just investigate during business hours.
+
+    # Login throttle — magic-link rate limiter denial counter. Noisier than a
+    # page; fires WARNING on >10 events in 10m so the operator can correlate
+    # against a /auth/login spike (legitimate users locked out by NAT, dashboard
+    # retry bug, bot sweep).
+    - name: instant-api-magic-link
+      rules:
+        - alert: MagicLinkEmailRateLimited
+          expr: |
+            sum(rate(instant_magic_link_email_rate_limited_total[10m])) * 600 > 10
+          for: 10m
+          labels:
+            severity: warning
+            service: api
+          annotations:
+            summary: "magic-link rate-limiter denied > 10 requests in 10m"
+            description: |
+              instant_magic_link_email_rate_limited_total > 10 events in 10m.
+              Per-email/per-IP magic-link throttle is denying real users at a
+              meaningful rate. Cross-check /auth/login traffic — a spike in
+              denials with no spike in requests means the dashboard is firing
+              retry storms; a spike in both means a bot sweep is wedging the
+              per-IP quota for legitimate users behind the same NAT.
+
+    # Provisioner circuit breaker state. 0=CLOSED (healthy), 1=HALF_OPEN
+    # (probing), 2=OPEN (fail-fast). Sustained OPEN means the provisioner
+    # stopped attempting calls to a backend after N consecutive failures —
+    # better than a hung call, but the customer flow is degraded.
+    - name: instant-provisioner-circuit
+      rules:
+        - alert: ProvisionerCircuitOpen
+          expr: |
+            max(instant_provisioner_circuit_state) by (backend) == 2
+          for: 5m
+          labels:
+            severity: critical
+            service: provisioner
+          annotations:
+            summary: "provisioner circuit breaker OPEN for backend={{ $labels.backend }}"
+            description: |
+              instant_provisioner_circuit_state{backend="{{ $labels.backend }}"} = 2
+              (OPEN) for >5m. The provisioner has stopped attempting calls to
+              this backend; all /db/new, /cache/new, /nosql/new, or /queue/new
+              requests targeting this backend are failing fast. Check the
+              backend's own /healthz, kubectl logs the upstream pod, and decide:
+              roll back the failing change, restart the backend pod, or wait
+              for the half-open probe to close the breaker naturally.
+
+        - alert: ProvisionerCircuitHalfOpen
+          expr: |
+            max(instant_provisioner_circuit_state) by (backend) == 1
+          for: 10m
+          labels:
+            severity: warning
+            service: provisioner
+          annotations:
+            summary: "provisioner circuit breaker HALF_OPEN sustained 10m for backend={{ $labels.backend }}"
+            description: |
+              Backend hasn't closed back to CLOSED after 10 minutes in
+              HALF_OPEN. Probes are still failing intermittently; investigate
+              the upstream before the breaker flips back to OPEN.
+
+    # Go-rendered email registry sentinel. After 2026-05-15 expiry-email retro
+    # the platform moved 18 email kinds off Brevo templates onto Go renderers.
+    # This counter ticks every time the worker tries to send an email kind
+    # that isn't in the eventEmailBuilders registry — a silent user-facing
+    # breakage that page-priority must be P0.
+    - name: instant-worker-email-renderer
+      rules:
+        - alert: EmailMissingRenderer
+          expr: |
+            sum(rate(email_missing_renderer_total[5m])) by (kind) > 0
+          for: 5m
+          labels:
+            severity: critical
+            service: worker
+          annotations:
+            summary: "email_missing_renderer kind={{ $labels.kind }} — silent user-facing breakage"
+            description: |
+              email_missing_renderer_total{kind="{{ $labels.kind }}"} > 0 for >5m.
+              The worker tried to send an email with a kind that has no Go
+              renderer registered in eventEmailBuilders — the message body
+              dropped silently on the floor. Add the renderer to
+              worker/internal/email/renderers/ AND add the kind to the
+              coverage test that iterates eventEmailBuilders so this can't
+              happen again for the same kind.
+
+    # /readyz deep-health gauge, exposed by api, worker, and provisioner via
+    # readyz_check_status{service,check}. 1=ok, 0.5=degraded, 0=failed. This
+    # rule mirrors the NR alert readyz-component-failed.json (which depends
+    # on the OTLP forwarder) so a Prometheus-only operator gets the same
+    # coverage. Same thresholds. Same labels.
+    - name: instant-readyz
+      rules:
+        - alert: ReadyzCheckFailed
+          expr: |
+            max(readyz_check_status) by (service, check) == 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "{{ $labels.service }} /readyz check {{ $labels.check }} is FAILED"
+            description: |
+              readyz_check_status{service="{{ $labels.service }}",check="{{ $labels.check }}"} == 0 for >5m.
+              The upstream this check probes is unreachable or rejecting calls.
+              Open the failing pod's /readyz body for last_error, cross-check
+              against the upstream status page, rotate keys if 401, file an
+              incident if 5xx.
+
+        - alert: ReadyzCheckDegraded
+          expr: |
+            max(readyz_check_status) by (service, check) == 0.5
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $labels.service }} /readyz check {{ $labels.check }} is DEGRADED"
+            description: |
+              readyz_check_status{service="{{ $labels.service }}",check="{{ $labels.check }}"} == 0.5 for >10m.
+              Upstream reachable but creds/policy off (the exact pattern that
+              would have caught the 2026-05-20 Brevo silent-rejection bug weeks
+              earlier — Brevo /v3/account returns 401 on every probe and the
+              brevo check goes 0.5). Rotate the credential / re-validate the
+              policy before the breaker escalates to failed.
+
+    # Brevo RECEIVE-side delivery-ratio alert (Prom mirror of NR
+    # email-delivery-ratio-low.json). Webhook receiver counter
+    # brevo_webhook_events_total{event} — `delivered` / total over 1h.
+    # The 2026-05-20 sender-domain dropout would have shown here within an
+    # hour as a falling ratio. Closed-set event labels keep cardinality
+    # bounded.
+    - name: instant-api-brevo-webhook
+      rules:
+        - alert: BrevoDeliveryRatioLow
+          expr: |
+            (
+              sum(rate(brevo_webhook_events_total{event="delivered"}[1h]))
+            )
+            /
+            clamp_min(
+              sum(rate(brevo_webhook_events_total{event=~"delivered|bounced_hard|bounced_soft|rejected|complaint|deferred|unsubscribed|error"}[1h])),
+              1e-9
+            )
+            < 0.95
+          for: 1h
+          labels:
+            severity: critical
+            service: api
+          annotations:
+            summary: "Brevo delivery ratio < 95% over 1h"
+            description: |
+              <95% of webhook-confirmed Brevo POSTs are landing as `delivered`.
+              Common causes: sender domain dropped from validated list (the
+              2026-05-20 launch incident), DKIM record stale, IP on a blocklist,
+              or a broad bounce/rejected event class spiking.
+
+        - alert: BrevoDeliveryRatioWarn
+          expr: |
+            (
+              sum(rate(brevo_webhook_events_total{event="delivered"}[1h]))
+            )
+            /
+            clamp_min(
+              sum(rate(brevo_webhook_events_total{event=~"delivered|bounced_hard|bounced_soft|rejected|complaint|deferred|unsubscribed|error"}[1h])),
+              1e-9
+            )
+            < 0.98
+          for: 1h
+          labels:
+            severity: warning
+            service: api
+          annotations:
+            summary: "Brevo delivery ratio < 98% over 1h (early warning)"
+
+    # Make-good worklist signal — billing.charge_undeliverable. Customer paid
+    # via Razorpay, platform could NOT translate the charge into a delivered
+    # upgrade (team unresolvable, or resolved plan tier not in plans.yaml).
+    # Each occurrence is a real customer charged but not upgraded; operator
+    # must reconcile in the Razorpay dashboard.
+    - name: instant-worker-billing
+      rules:
+        - alert: BillingChargeUndeliverable
+          expr: |
+            sum(rate(instant_billing_charge_undeliverable_total[1h])) > 0
+          for: 5m
+          labels:
+            severity: critical
+            service: worker
+          annotations:
+            summary: "billing.charge_undeliverable > 0 — paid customer not upgraded"
+            description: |
+              instant_billing_charge_undeliverable_total > 0 in the last hour.
+              A subscription.charged Razorpay webhook confirmed a real card
+              charge that the platform could NOT translate into a delivered
+              upgrade. Find the team in audit_log where audit_kind = 'billing.charge_undeliverable',
+              cross-check against the Razorpay dashboard, refund or hand-grant
+              the tier.

newrelic/alerts/brevo-send-errors-spike.json

@@ -0,0 +1,38 @@
+{
+  "name": "instant-worker — brevo_send_errors_total spike (SEND-side of email pipeline broken)",
+  "type": "NRQL",
+  "description": "P1 page. The SEND-side complement to email-delivery-ratio-low.json: that alert measures Brevo → /webhooks/brevo (RECEIVE), this one measures worker → Brevo /v3/smtp/email (SEND). Brevo's transactional API returns 201 the instant it accepts the POST, so a degraded send path (auth failure, rate limit, network) NEVER affects the delivery-ratio alert — the messages simply never queue at Brevo. The classification label separates fixable-by-key-rotation (permanent: 401/403) from fixable-by-backoff (transient: 5xx, 429) so the operator knows which runbook to follow. Source: worker/internal/metrics/metrics.go (BrevoSendErrorsTotal), emit site worker/internal/email/brevo/brevo.go.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT sum(brevo_send_errors_total) FROM Metric WHERE metricName = 'brevo_send_errors_total' FACET classification, status_code"
+  },
+  "terms": [
+    {
+      "priority": "CRITICAL",
+      "operator": "ABOVE",
+      "threshold": 10,
+      "thresholdDuration": 600,
+      "thresholdOccurrences": "ALL"
+    },
+    {
+      "priority": "WARNING",
+      "operator": "ABOVE",
+      "threshold": 3,
+      "thresholdDuration": 600,
+      "thresholdOccurrences": "ALL"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 60,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 3600,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/email-missing-renderer.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-worker — email_missing_renderer (no Go renderer for an email kind — silent user-facing breakage)",
+  "type": "NRQL",
+  "description": "P0 page. email_missing_renderer_total{kind} ticks every time the worker tries to send an email whose `kind` is not in the eventEmailBuilders registry. After the 2026-05-15 expiry-email retro the platform moved 18 email kinds off Brevo templates and onto Go-side renderers; this metric is the registry-iteration sentinel for that decision. ANY tick means a customer-facing email path is broken silently — no body content, no fallback, just a worker ERROR log + a missed message. Runbook: search NR Logs for `worker.email.missing_renderer` with the offending kind, then add the renderer to worker/internal/email/renderers/ AND add the kind to the coverage test that iterates eventEmailBuilders. Source: worker/internal/metrics/metrics.go (EmailMissingRendererTotal), emit site worker/internal/email/forwarder.go.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT sum(email_missing_renderer_total) FROM Metric WHERE metricName = 'email_missing_renderer_total' FACET kind"
+  },
+  "terms": [
+    {
+      "priority": "CRITICAL",
+      "operator": "ABOVE",
+      "threshold": 0,
+      "thresholdDuration": 300,
+      "thresholdOccurrences": "AT_LEAST_ONCE"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 60,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 3600,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/magic-link-email-rate-limited.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-api — magic_link_email_rate_limited spike (legitimate user being throttled)",
+  "type": "NRQL",
+  "description": "P2 warning. instant_magic_link_email_rate_limited_total ticks every time the magic-link rate limiter denies a /auth/login request (per-email or per-IP). Modal harmless cause: a user double-clicking the login button. Modal harmful cause: an upstream bot sweep wedging the per-IP quota for legitimate users sharing the same NAT or office network, OR a bug in the dashboard re-firing the magic-link request on a hot retry path (BugBash 2026-05-20 found one such case). Threshold here is intentionally noisier than a page — fire WARNING on >10 events in 10 minutes so the operator can correlate against a /auth/login spike. Source: api/internal/metrics/metrics.go (MagicLinkEmailRateLimitedTotal), emit site api/internal/handlers/auth.go.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT sum(instant_magic_link_email_rate_limited_total) FROM Metric WHERE metricName = 'instant_magic_link_email_rate_limited_total'"
+  },
+  "terms": [
+    {
+      "priority": "WARNING",
+      "operator": "ABOVE",
+      "threshold": 10,
+      "thresholdDuration": 600,
+      "thresholdOccurrences": "ALL"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 60,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 3600,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/orphan-sweep-no-db-row.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-worker — orphan_sweep reaped instant-deploy-* ns with no backing deployments row (P0-3 atomic-provision)",
+  "type": "NRQL",
+  "description": "P0 page. instant_orphan_sweep_reaped_total{reason='no_db_row'} > 0 sustained for >1h. A no_db_row event means the orphan_sweep reconciler found an instant-deploy-<appID> namespace in the live cluster with NO matching deployments row in the platform DB. That can only happen if the api created the k8s namespace but the INSERT into deployments never committed — the P0-3 atomic-provision symptom surfacing in prod. Runbook: search NR Logs for `jobs.orphan_sweep.proposed_reap` with reason=no_db_row, capture the app_id, then trace back through the api POST /deploy/new logs for the same time window to find the partial-commit path that needs the atomic-rollback fix. Source: worker/internal/metrics/metrics.go (OrphanSweepReapedTotal), emit site worker/internal/jobs/orphan_sweep_reconciler.go.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT sum(instant_orphan_sweep_reaped_total) FROM Metric WHERE metricName = 'instant_orphan_sweep_reaped_total' AND reason = 'no_db_row'"
+  },
+  "terms": [
+    {
+      "priority": "CRITICAL",
+      "operator": "ABOVE",
+      "threshold": 0,
+      "thresholdDuration": 3600,
+      "thresholdOccurrences": "AT_LEAST_ONCE"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 300,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 7200,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}