observability: synthetic monitors + silent-failure alerts for billing/funnel/pods

Adds NR Synthetics monitors + NR Alert conditions so the class of silent
production failures surfaced this session gets caught automatically: a dead
paid funnel, an unhandled Razorpay webhook event, and a charged-but-not-
upgraded customer — all of which broke prod with nothing alerting.

Synthetic monitors (newrelic/synthetics/, applied by synthetics-apply.sh):
- instant-api healthz ping            — SIMPLE  /healthz, 5m
- instanode marketing home ping       — SIMPLE  instanode.dev/, 5m
- instanode dashboard login page      — BROWSER /app SPA mount, 5m
- instant-api openapi.json ping       — SIMPLE  /openapi.json, 15m
- instant-api healthz scripted        — SCRIPT_API: asserts ok=true,
  non-placeholder commit_id, migration_status=ok, build_time <= 30d
  (build-SHA freshness — catches stale-image deploys)

Alert conditions (newrelic/alerts/, applied by apply.sh onto the existing
'instant-api alerts' policy) — all key on signals the app already emits, no
new app code:
- billing-charge-undeliverable        CRIT  audit_kind=billing.charge_undeliverable, any occurrence
- billing-webhook-unhandled-event     WARN  message=billing.webhook.unhandled_event, any occurrence
- billing-reconciler-orphan-corrected WARN  instant_billing_reconciler_orphan_corrected_total > 0 / 15m
- entitlement-drift-corrected         WARN/CRIT instant_entitlement_drift_corrected_total > 3/20 per 1h
- zero-upgrades-7d                    CRIT  subscription.upgraded == 0 sustained 7d (dead funnel)
- checkout-failure-spike              WARN/CRIT checkout ERROR logs > 3/15 per 1h
- api-healthz-down                    WARN/CRIT SyntheticCheck result=FAILED
- backend-service-no-logs             CRIT  api/worker/provisioner zero logs 10m

Elevated api 5xx is already covered by the pre-existing api-5xx-rate-high
condition — no duplicate added.

synthetics-apply.sh is a companion to apply.sh (separate NerdGraph mutation
family for synthetic monitors); both are idempotent by name. Full monitor +
alert table, thresholds, and apply commands in newrelic/synthetics/SYNTHETICS.md.

Definitions only — not applied to NR (no API key available in this
environment); operator applies via ./apply.sh + ./synthetics-apply.sh against
account 7958263.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: mastermanas805

newrelic/README.md

@@ -144,3 +144,17 @@ Alert conditions (4):
 - `instant-api — p95 latency > 500ms (5m)`
 - `instant-api — NATS connection failures`
 - `instant-worker — no jobs processed in 10m`
+
+## Synthetic monitoring + silent-failure alerting
+
+Synthetic monitors (NR Synthetics) live in `synthetics/` and are applied by
+`synthetics-apply.sh` (companion to `apply.sh`, separate because synthetic
+monitors use a different family of NerdGraph mutations). Eight new alert
+conditions in `alerts/` catch silent billing/funnel/pod failures keyed on
+signals the app already emits — `billing.charge_undeliverable`,
+`billing.webhook.unhandled_event`, `instant_billing_reconciler_orphan_corrected_total`,
+`instant_entitlement_drift_corrected_total`, `subscription.upgraded`,
+checkout error logs, `SyntheticCheck` results, and per-service log heartbeat.
+
+See `synthetics/SYNTHETICS.md` for the full monitor + alert table, thresholds,
+rationale, and apply commands.

newrelic/alerts/api-healthz-down.json

@@ -0,0 +1,38 @@
+{
+  "name": "instant-api — /healthz synthetic failing (api pod down / unreachable)",
+  "type": "NRQL",
+  "description": "Page when the api-healthz synthetic monitor records a failed check — the api is down, unreachable, or its /healthz endpoint is returning non-2xx. Keyed on the SyntheticCheck event the New Relic Synthetics ping + scripted-API monitors emit (result = 'FAILED'). Catches: api pod crash loop missed by k8s readiness probes, ingress/TLS breakage, full-cluster outage. CRITICAL on 2 consecutive failed checks (~10m at the 5-min monitor period) to ride out a single transient blip. Source: SyntheticCheck events from monitors 'instant-api healthz ping' and 'instant-api healthz scripted (build-SHA freshness)' — see newrelic/synthetics/. If the NR k8s integration is later installed, add a companion K8sContainerSample condition for pod-restart-count.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT count(*) FROM SyntheticCheck WHERE monitorName LIKE 'instant-api healthz%' AND result = 'FAILED'"
+  },
+  "terms": [
+    {
+      "priority": "CRITICAL",
+      "operator": "ABOVE_OR_EQUALS",
+      "threshold": 2,
+      "thresholdDuration": 600,
+      "thresholdOccurrences": "ALL"
+    },
+    {
+      "priority": "WARNING",
+      "operator": "ABOVE_OR_EQUALS",
+      "threshold": 1,
+      "thresholdDuration": 300,
+      "thresholdOccurrences": "ALL"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 300,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 180,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 1800,
+    "openViolationOnExpiration": true,
+    "closeViolationsOnExpiration": false
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/backend-service-no-logs.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant — backend service emitted zero logs in 10m (pod down: api / worker / provisioner)",
+  "type": "NRQL",
+  "description": "Page when any of the three backend services (api, worker, provisioner) emits ZERO log records for 10 minutes — a per-service liveness check that catches a crashed/evicted pod, a deadlocked process, or a k8s rollout that never came healthy, even when the k8s readiness probe was satisfied. Faceted by service so the violation names which pod went dark. Every backend binary logs continuously under normal operation (request logs, River job middleware, gRPC handler logs), so a 10-minute silence on any one service is a real outage. Source: Log records, FACET service. Complements api-healthz-down (which checks the api from outside via Synthetics) by covering worker + provisioner, which have no public HTTP surface to ping. fillValue STATIC 0 ensures a fully silent stream is treated as zero, not no-data.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT count(*) FROM Log WHERE service IN ('api', 'worker', 'provisioner') FACET service"
+  },
+  "terms": [
+    {
+      "priority": "CRITICAL",
+      "operator": "BELOW",
+      "threshold": 1,
+      "thresholdDuration": 600,
+      "thresholdOccurrences": "ALL"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 60,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 1200,
+    "openViolationOnExpiration": true,
+    "closeViolationsOnExpiration": false
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/billing-charge-undeliverable.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-api — billing.charge_undeliverable [customer charged, upgrade NOT delivered]",
+  "type": "NRQL",
+  "description": "HIGHEST SEVERITY. Page on ANY occurrence of a billing.charge_undeliverable audit event — a Razorpay subscription.charged webhook confirmed a real card charge that the platform could NOT translate into a delivered upgrade (team unresolvable, or resolved plan tier not in plans.yaml). Each occurrence is a real customer who paid and did not get what they paid for; an operator must reconcile the charge in the Razorpay dashboard (refund or hand-grant the tier). This is the make-good worklist signal. Source: audit_log row audit_kind = 'billing.charge_undeliverable' written by emitChargeUndeliverableAudit in api/internal/handlers/billing.go, paired with a loud slog.Error. The audit row surfaces in NR as a Log record with the audit_kind attribute (see api/internal/models/audit_log.go — the field is deliberately 'audit_kind', not 'kind'). Threshold is ABOVE 0 so a single charged-but-not-upgraded customer pages immediately.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT count(*) FROM Log WHERE service = 'api' AND audit_kind = 'billing.charge_undeliverable'"
+  },
+  "terms": [
+    {
+      "priority": "CRITICAL",
+      "operator": "ABOVE",
+      "threshold": 0,
+      "thresholdDuration": 300,
+      "thresholdOccurrences": "AT_LEAST_ONCE"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 60,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 3600,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/billing-reconciler-orphan-corrected.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-api — billing reconciler orphan-corrected > 0 (15m) [paid customer slipped past primary path]",
+  "type": "NRQL",
+  "description": "Warn when the worker's billing reconciler orphan sweep has to correct a paid customer the primary webhook path missed. instant_billing_reconciler_orphan_corrected_total counts teams the reconciler elevated because they paid at Razorpay but had no persisted subscription_id (the pending_checkouts row never got its subscription_id written — a webhook-handling or DB-write gap). The reconciler is the safety net; every correction it makes means a customer would have been charged-but-not-upgraded if the net were not there. A rising count says the PRIMARY path (synchronous webhook → upgrade) is dropping payments and should be investigated. Source: worker/internal/metrics/metrics.go BillingReconcilerOrphanCorrected counter, scraped into NR as a Metric. Threshold ABOVE 0 over 15m (one+ reconciler tick); WARNING because the reconciler already self-corrected — the customer IS upgraded — but the primary-path gap needs a root-cause fix.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT sum(instant_billing_reconciler_orphan_corrected_total) FROM Metric WHERE service = 'worker'"
+  },
+  "terms": [
+    {
+      "priority": "WARNING",
+      "operator": "ABOVE",
+      "threshold": 0,
+      "thresholdDuration": 900,
+      "thresholdOccurrences": "AT_LEAST_ONCE"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 300,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 180,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 1800,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/billing-webhook-unhandled-event.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-api — billing.webhook.unhandled_event [Razorpay event silently swallowed]",
+  "type": "NRQL",
+  "description": "Warn on ANY occurrence of a billing.webhook.unhandled_event WARN log. The Razorpay webhook dispatcher's `default:` case used to silently swallow event types the handler did not recognise — a coverage gap that hid, for example, recurring-payment events the platform never acted on. The default case now logs at WARN with message = 'billing.webhook.unhandled_event' and the event_type / event_id attributes (api/internal/handlers/billing.go ~line 1132). This alert makes that WARN visible: any unhandled Razorpay event type means a webhook coverage gap to triage — either a new event we should handle, or noise to explicitly ignore. Threshold ABOVE 0 (single occurrence) — these are rare and each one is a real gap. Not CRITICAL because a swallowed event is not always customer-impacting, but it must never go unnoticed again.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT count(*) FROM Log WHERE service = 'api' AND message = 'billing.webhook.unhandled_event'"
+  },
+  "terms": [
+    {
+      "priority": "WARNING",
+      "operator": "ABOVE",
+      "threshold": 0,
+      "thresholdDuration": 300,
+      "thresholdOccurrences": "AT_LEAST_ONCE"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 60,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 3600,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/checkout-failure-spike.json

@@ -0,0 +1,38 @@
+{
+  "name": "instant-api — checkout failure / subscription.pending spike (1h)",
+  "type": "NRQL",
+  "description": "Warn when POST /api/v1/billing/checkout is failing — the symptom that the prod paid funnel was broken this session ('seller does not support recurring payments': Razorpay rejected the subscription create, so checkouts errored or hung in subscription.pending and never charged). Catches a checkout that breaks BEFORE any charge webhook would fire — earlier in the funnel than billing.charge_undeliverable, which only sees post-charge failures. Signal: ERROR-level Log records from the checkout handler (message contains 'checkout'). A sustained non-trivial error rate means the buy button is broken for real customers. WARNING at >3/h, CRITICAL at >15/h — low absolute thresholds because at current volume even a handful of failed checkouts is most of the funnel. Source: api/internal/handlers/billing.go checkout handler error logs (service='api'). Pairs with the billing-dunning dashboard's 'Checkout failure rate by error.code' widget for the per-code breakdown.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT count(*) FROM Log WHERE service = 'api' AND level = 'ERROR' AND message LIKE '%checkout%'"
+  },
+  "terms": [
+    {
+      "priority": "WARNING",
+      "operator": "ABOVE",
+      "threshold": 3,
+      "thresholdDuration": 3600,
+      "thresholdOccurrences": "ALL"
+    },
+    {
+      "priority": "CRITICAL",
+      "operator": "ABOVE",
+      "threshold": 15,
+      "thresholdDuration": 3600,
+      "thresholdOccurrences": "ALL"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 300,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 120,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 3600,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/entitlement-drift-corrected.json

@@ -0,0 +1,38 @@
+{
+  "name": "instant-api — entitlement drift-corrected rising (1h) [resources running on wrong tier limits]",
+  "type": "NRQL",
+  "description": "Warn when the worker's entitlement reconciler is correcting infra entitlement drift — resources whose live connection cap / storage limit no longer matches what their team's tier should grant. instant_entitlement_drift_corrected_total counts resources the reconciler actually re-graded back to the correct limits (a Postgres ALTER USER / Redis maxmemory change really happened). A sustained non-zero rate means upgrades/downgrades are NOT propagating their entitlement changes synchronously — paying customers may be stuck on lower limits, or downgraded customers retaining higher limits. The reconciler self-heals, but a rising count points at a broken primary entitlement-propagation path (ElevateResourceTiersByTeam / provisioner RegradeResource). Source: worker/internal/metrics/metrics.go EntitlementDriftCorrected counter, scraped into NR as a Metric. WARNING — the reconciler corrects it, but the gap needs root-cause work. Threshold ABOVE 3 corrections in a rolling 1h window so steady-state noise (one stray row) does not page.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT sum(instant_entitlement_drift_corrected_total) FROM Metric WHERE service = 'worker'"
+  },
+  "terms": [
+    {
+      "priority": "WARNING",
+      "operator": "ABOVE",
+      "threshold": 3,
+      "thresholdDuration": 3600,
+      "thresholdOccurrences": "ALL"
+    },
+    {
+      "priority": "CRITICAL",
+      "operator": "ABOVE",
+      "threshold": 20,
+      "thresholdDuration": 3600,
+      "thresholdOccurrences": "ALL"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 300,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 180,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 7200,
+    "openViolationOnExpiration": false,
+    "closeViolationsOnExpiration": true
+  },
+  "violationTimeLimitSeconds": 86400
+}

newrelic/alerts/zero-upgrades-7d.json

@@ -0,0 +1,31 @@
+{
+  "name": "instant-api — zero successful upgrades in 7 days [dead paid funnel]",
+  "type": "NRQL",
+  "description": "Page when NOT A SINGLE successful paid upgrade has been recorded for a sustained 7-day window. This is the dead-funnel catch: it would have fired the day the prod paid funnel went silent ('seller does not support recurring payments' — checkouts created, no charge webhook ever arrived, nothing else alerted). 'Success' = a subscription.upgraded audit event, which fires only when a Razorpay subscription.charged webhook actually moves a team to a higher tier (api/internal/models/audit_kinds.go AuditKindSubscriptionUpgraded). A low-baseline static threshold is used rather than a NR Baseline condition because early-stage upgrade volume is too sparse for anomaly math — at this stage ANY 7-day stretch with zero upgrades is itself the anomaly worth a human looking. thresholdDuration is 7 days (604800s); fillValue STATIC 0 so a totally silent stream still counts as zero rather than producing no data. RE-TUNE: once steady upgrade volume exists, replace with a Baseline condition keyed on '< Nx expected'.",
+  "enabled": true,
+  "nrql": {
+    "query": "SELECT count(*) FROM Log WHERE service = 'api' AND audit_kind = 'subscription.upgraded'"
+  },
+  "terms": [
+    {
+      "priority": "CRITICAL",
+      "operator": "BELOW_OR_EQUALS",
+      "threshold": 0,
+      "thresholdDuration": 604800,
+      "thresholdOccurrences": "ALL"
+    }
+  ],
+  "signal": {
+    "aggregationWindow": 3600,
+    "aggregationMethod": "EVENT_FLOW",
+    "aggregationDelay": 600,
+    "fillOption": "STATIC",
+    "fillValue": 0
+  },
+  "expiration": {
+    "expirationDuration": 86400,
+    "openViolationOnExpiration": true,
+    "closeViolationsOnExpiration": false
+  },
+  "violationTimeLimitSeconds": 259200
+}