fix(observability): reconcile NR alerts/dashboards + Prometheus to real fields

BugBash 2026-05-18 findings P1-W3-01..05, P1-W5-08, T13-S5-05.

NR audit field normalization (P1-W3-01/02): the api now emits the audit
kind under the slog field `audit_kind`. Repoint every audit-driven alert
and dashboard query that used the bare `kind` field:
  alerts:     paused-resource-stale, deploy-time-degraded,
              promote-bypass-detected, grace-terminated-spike,
              deploy-failure-rate-high, admin-allowlist-breach (desc)
  dashboards: resource-lifecycle, admin-defense, billing-dunning,
              promote-approval, ops-rollup

deploy-time-degraded + resource-lifecycle + ops-rollup (P1-W3-02): the
code emits `time_to_healthy_seconds`, not `deploy_ready_seconds`.

worker-stalled alert + worker dashboard (P1-W3-03): the worker emits
`jobs.middleware.work_ok` / `jobs.middleware.work_failed` (not
`job.completed` / `job.failed`), with the job name under `kind`.
Repointed every worker widget + the stall alert; facet on `kind`.

storage-iam-create-failed (P1-W3-05): the non-existent timeslice metric
`Custom/Storage/IamUsersFailed/create` is replaced by the real Prometheus
counter `instant_storage_iam_users_failed_total{op="create"}`. Added the
guaranteed-working equivalent rule StorageIamCreateFailures to
k8s/prometheus-rules.yaml.

prometheus.yml (T13-S5-05): `instant-api-k8s` targeted the NodePort
retired 2026-05-11. Repointed to the ClusterIP Service DNS (job renamed
`instant-api` to match the up{} selector in prometheus-rules.yaml) and
added an `instant-worker` scrape job so AnonymousResourceExpiryStopped
(worker-only metrics) can fire.

worker /metrics scrape (P1-W5-08): added prometheus.io/scrape pod
annotations and a headless ClusterIP `instant-worker` Service on 8091
so the new scrape job resolves.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: mastermanas805

k8s/prometheus-rules.yaml

@@ -85,3 +85,17 @@ spec:
             severity: warning
           annotations:
             summary: "No anonymous expirations in 1h while many active TTL resources remain (worker metrics)"
+
+        # Storage IAM user creation failures in the /storage/new hot path.
+        # A failure here blocks a paying customer's signup-to-first-byte flow.
+        # Counter source: api/internal/handlers/storage.go (instant_storage_iam_users_failed_total,
+        # labelled op=create|delete). This is the guaranteed-working equivalent of the
+        # New Relic storage-iam-create-failed alert, which depends on the NR Prometheus
+        # OpenMetrics integration being configured.
+        - alert: StorageIamCreateFailures
+          expr: increase(instant_storage_iam_users_failed_total{op="create"}[5m]) > 0
+          for: 0m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Storage IAM user create failures > 0 in 5m (signup-blocking)"

k8s/worker/deployment.yaml

@@ -22,6 +22,15 @@ spec:
     metadata:
       labels:
         app: instant-worker
+      annotations:
+        # Prometheus pod-annotation discovery (P1-W5-08). The worker serves
+        # /metrics on port 8091 — the same port as /healthz. These let an
+        # annotation-based Prometheus discovery scrape the pod directly; the
+        # in-cluster instant-worker Service below is the static_configs path
+        # used by infra/prometheus/prometheus.yml.
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "8091"
+        prometheus.io/path: "/metrics"
     spec:
       serviceAccountName: instant-worker
       # Multi-worker is safe — River queue uses Postgres SELECT FOR UPDATE
@@ -40,8 +49,11 @@ spec:
           image: instant-worker:local
           imagePullPolicy: Never
           ports:
+            # 8091 serves both /healthz (probes) and /metrics (Prometheus
+            # scrape). Named 'metrics' so the instant-worker Service below
+            # can reference it by name.
             - containerPort: 8091
-              name: liveness
+              name: metrics
           # Probes (2026-05-18): the worker binary registers ONLY /healthz
           # (plus /metrics) — it never serves /livez. All three probes hit
           # /healthz. /healthz checks River queue connectivity + DB pool
@@ -149,3 +161,31 @@ spec:
               value: "noreply@instanode.dev"
             - name: BREVO_SENDER_NAME
               value: "instanode"
+---
+# Headless ClusterIP Service for the worker (P1-W5-08 / T13-S5-05). The worker
+# is a background-job processor with no inbound API traffic, but it serves
+# /healthz + /metrics on 8091. This Service exists solely so Prometheus can
+# resolve instant-worker.instant-infra.svc.cluster.local:8091 — the target of
+# the 'instant-worker' scrape job in infra/prometheus/prometheus.yml, which
+# feeds the AnonymousResourceExpiryStopped + StorageIamCreateFailures alerts.
+apiVersion: v1
+kind: Service
+metadata:
+  name: instant-worker
+  namespace: instant-infra
+  labels:
+    app: instant-worker
+  annotations:
+    prometheus.io/scrape: "true"
+    prometheus.io/port: "8091"
+    prometheus.io/path: "/metrics"
+spec:
+  type: ClusterIP
+  clusterIP: None
+  selector:
+    app: instant-worker
+  ports:
+    - name: metrics
+      port: 8091
+      targetPort: metrics
+      protocol: TCP

newrelic/alerts/admin-allowlist-breach.json

@@ -1,7 +1,7 @@
 {
   "name": "instant-api — admin.access from non-allowlist user",
   "type": "NRQL",
-  "description": "CRITICAL. Fires on any admin.access audit event where the ADMIN_EMAILS allowlist middleware rejected the caller (denied_by = 'allowlist_miss') — i.e. a user who is NOT on the allowlist reached an admin route and tripped the allowlist gate. Sustained hits mean someone with a real signed-in session is probing the admin surface. The audit middleware (api/internal/middleware/admin_audit.go) writes denied_by = 'allowlist_miss' into every admin.access event whose 403 came from the allowlist check (vs 'rate_limit' for the rate-limiter, '' for success). FIELD NOTE: this query was previously written against a non-existent `allowed = false` tag; the middleware emits `denied_by`, not `allowed`. FOLLOW-UP: to also catch a true fail-OPEN (non-allowlist user getting a 200, not a 403), the api should emit an explicit `is_allowlisted` boolean on every admin.access event — denied_by only covers the correctly-denied case. Source: audit_log kind = 'admin.access' emitted by the admin audit middleware (PR #66+).",
+  "description": "CRITICAL. Fires on any admin.access audit event where the ADMIN_EMAILS allowlist middleware rejected the caller (denied_by = 'allowlist_miss') — i.e. a user who is NOT on the allowlist reached an admin route and tripped the allowlist gate. Sustained hits mean someone with a real signed-in session is probing the admin surface. The audit middleware (api/internal/middleware/admin_audit.go) writes denied_by = 'allowlist_miss' into every admin.access event whose 403 came from the allowlist check (vs 'rate_limit' for the rate-limiter, '' for success). FIELD NOTE: this query was previously written against a non-existent `allowed = false` tag; the middleware emits `denied_by`, not `allowed`. FOLLOW-UP: to also catch a true fail-OPEN (non-allowlist user getting a 200, not a 403), the api should emit an explicit `is_allowlisted` boolean on every admin.access event — denied_by only covers the correctly-denied case. Source: audit_log audit_kind = 'admin.access' emitted by the admin audit middleware (PR #66+).",
   "enabled": true,
   "nrql": {
     "query": "SELECT count(*) FROM Log WHERE service = 'api' AND audit_kind = 'admin.access' AND denied_by = 'allowlist_miss'"

newrelic/alerts/deploy-failure-rate-high.json

@@ -1,10 +1,10 @@
 {
   "name": "instant-api — deploy failure rate > 30% (1h)",
   "type": "NRQL",
-  "description": "CRITICAL when deploy failures exceed 30% of created deploys over the last hour — build-infra degradation signal (Kaniko OOMs, registry pulls failing, k8s scheduler stuck, ingress-cert provisioning broken). Pairs with the deploy success-rate widget in the resource-lifecycle dashboard. Source: audit_log kind IN ('deploy.created', 'deploy.failed') emitted by api/internal/handlers/deploy.go.",
+  "description": "CRITICAL when deploy failures exceed 30% of created deploys over the last hour — build-infra degradation signal (Kaniko OOMs, registry pulls failing, k8s scheduler stuck, ingress-cert provisioning broken). Pairs with the deploy success-rate widget in the resource-lifecycle dashboard. Source: audit_log audit_kind IN ('deploy.created', 'deploy.failed') emitted by api/internal/handlers/deploy.go.",
   "enabled": true,
   "nrql": {
-    "query": "SELECT (filter(uniqueCount(deploy_id), WHERE kind = 'deploy.failed') / filter(uniqueCount(deploy_id), WHERE kind = 'deploy.created')) * 100 FROM Log WHERE service = 'api' AND kind IN ('deploy.created', 'deploy.failed')"
+    "query": "SELECT (filter(uniqueCount(deploy_id), WHERE audit_kind = 'deploy.failed') / filter(uniqueCount(deploy_id), WHERE audit_kind = 'deploy.created')) * 100 FROM Log WHERE service = 'api' AND audit_kind IN ('deploy.created', 'deploy.failed')"
   },
   "terms": [
     {

newrelic/alerts/deploy-time-degraded.json

@@ -1,10 +1,10 @@
 {
   "name": "instant-api — median deploy time > 5 min (build infra degradation)",
   "type": "NRQL",
-  "description": "WARN when the median time from deploy.created to deploy.healthy exceeds 5 minutes over a rolling 1h window. A healthy nginx-tarball baseline is ~30s; >5min sustained means Kaniko build is degraded, the image registry is slow, or k8s pod scheduling is stuck. Source: audit_log kind = 'deploy.healthy' with `deploy_ready_seconds` field, emitted by api/internal/handlers/deploy.go.",
+  "description": "WARN when the median time from deploy.created to deploy.healthy exceeds 5 minutes over a rolling 1h window. A healthy nginx-tarball baseline is ~30s; >5min sustained means Kaniko build is degraded, the image registry is slow, or k8s pod scheduling is stuck. Source: audit_log audit_kind = 'deploy.healthy' with `time_to_healthy_seconds` field, emitted by api/internal/handlers/deploy.go.",
   "enabled": true,
   "nrql": {
-    "query": "SELECT percentile(deploy_ready_seconds, 50) FROM Log WHERE service = 'api' AND kind = 'deploy.healthy' AND deploy_ready_seconds IS NOT NULL"
+    "query": "SELECT percentile(time_to_healthy_seconds, 50) FROM Log WHERE service = 'api' AND audit_kind = 'deploy.healthy' AND time_to_healthy_seconds IS NOT NULL"
   },
   "terms": [
     {

newrelic/alerts/grace-terminated-spike.json

@@ -1,10 +1,10 @@
 {
   "name": "instant-api — payment.grace_terminated spike > 5x rolling 7d avg",
   "type": "NRQL",
-  "description": "WARN when grace_terminated events in the last hour exceed 5x the rolling 7-day per-hour average — mass payment-method failure signal (PSP outage, expired-card wave, fraud-block, etc.). NRQL static thresholds can't natively reference rolling averages, so this fires on absolute hourly count; the threshold is set from the expected per-hour 7d baseline (operator should re-tune after 30d of data). For a true ratio alert, use a Baseline-condition variant once the surface has steady traffic. Source: audit_log kind = 'payment.grace_terminated' emitted by api/internal/handlers/billing.go (PR #66 era).",
+  "description": "WARN when grace_terminated events in the last hour exceed 5x the rolling 7-day per-hour average — mass payment-method failure signal (PSP outage, expired-card wave, fraud-block, etc.). NRQL static thresholds can't natively reference rolling averages, so this fires on absolute hourly count; the threshold is set from the expected per-hour 7d baseline (operator should re-tune after 30d of data). For a true ratio alert, use a Baseline-condition variant once the surface has steady traffic. Source: audit_log audit_kind = 'payment.grace_terminated' emitted by api/internal/handlers/billing.go (PR #66 era).",
   "enabled": true,
   "nrql": {
-    "query": "SELECT count(*) FROM Log WHERE service = 'api' AND kind = 'payment.grace_terminated'"
+    "query": "SELECT count(*) FROM Log WHERE service = 'api' AND audit_kind = 'payment.grace_terminated'"
   },
   "terms": [
     {

newrelic/alerts/paused-resource-stale.json

@@ -1,10 +1,10 @@
 {
   "name": "instant-api — resource paused > 30d (should be terminated)",
   "type": "NRQL",
-  "description": "WARN when any resource is still in paused state more than 30 days after its pause event with no matching resume. Paused resources still consume disk and the platform DB row; 30d+ paused indicates the lifecycle policy should have terminated them. The query relies on a `paused_age_days` field stamped by the worker's resource-lifecycle job at scan time; alert fires if any resource exceeds the threshold during the most recent scan. Source: audit_log kind = 'resource.lifecycle_scan' (worker, migration 027 era).",
+  "description": "WARN when any resource is still in paused state more than 30 days after its pause event with no matching resume. Paused resources still consume disk and the platform DB row; 30d+ paused indicates the lifecycle policy should have terminated them. The query relies on a `paused_age_days` field stamped by the worker's resource-lifecycle job at scan time; alert fires if any resource exceeds the threshold during the most recent scan. Source: audit_log audit_kind = 'resource.lifecycle_scan' (worker, migration 027 era).",
   "enabled": true,
   "nrql": {
-    "query": "SELECT max(paused_age_days) FROM Log WHERE service = 'worker' AND kind = 'resource.lifecycle_scan' AND paused_age_days IS NOT NULL"
+    "query": "SELECT max(paused_age_days) FROM Log WHERE service = 'worker' AND audit_kind = 'resource.lifecycle_scan' AND paused_age_days IS NOT NULL"
   },
   "terms": [
     {

newrelic/alerts/promote-bypass-detected.json

@@ -1,10 +1,10 @@
 {
   "name": "instant-api — promote.approved without prior approval_requested (non-dev)",
   "type": "NRQL",
-  "description": "CRITICAL. Fires on any promote.approved audit_log event for env != 'development' that lacks a matching promote.approval_requested in the prior 24h. This indicates the email-link approval gate was bypassed — either the token was forged, the gate-check was disabled, or a code path skipped the request step. The middleware stamps `bypass_detected = true` when it cannot find a matching pending request at approval time. Source: audit_log kind = 'promote.approved' emitted by api/internal/handlers/stack.go (PR #65 era).",
+  "description": "CRITICAL. Fires on any promote.approved audit_log event for env != 'development' that lacks a matching promote.approval_requested in the prior 24h. This indicates the email-link approval gate was bypassed — either the token was forged, the gate-check was disabled, or a code path skipped the request step. The middleware stamps `bypass_detected = true` when it cannot find a matching pending request at approval time. Source: audit_log audit_kind = 'promote.approved' emitted by api/internal/handlers/stack.go (PR #65 era).",
   "enabled": true,
   "nrql": {
-    "query": "SELECT count(*) FROM Log WHERE service = 'api' AND kind = 'promote.approved' AND env != 'development' AND bypass_detected = true"
+    "query": "SELECT count(*) FROM Log WHERE service = 'api' AND audit_kind = 'promote.approved' AND env != 'development' AND bypass_detected = true"
   },
   "terms": [
     {

newrelic/alerts/storage-iam-create-failed.json

@@ -1,10 +1,10 @@
 {
   "name": "instant-api — storage IAM user create failures > 0 (5m) [signup-blocking]",
   "type": "NRQL",
-  "description": "Page when any storage IAM user creation fails. Storage IAM provisioning happens in the /storage/new hot path (W9-C1); a failure here means a new paying customer cannot get a usable storage bucket and the signup-to-first-byte path is broken. Source: the instant_storage_iam_users_failed_total Prometheus counter scraped into Metric, labelled by op (create | delete). The metric is exported by api/internal/handlers/storage.go.",
+  "description": "Page when any storage IAM user creation fails. Storage IAM provisioning happens in the /storage/new hot path (W9-C1); a failure here means a new paying customer cannot get a usable storage bucket and the signup-to-first-byte path is broken. Source: the instant_storage_iam_users_failed_total Prometheus counter exported by api/internal/handlers/storage.go on the api /metrics endpoint, labelled by op (create | delete). FIELD NOTE (2026-05-18, BugBash P1-W3-05): this query previously referenced the timeslice metric 'Custom/Storage/IamUsersFailed/create', which has zero emitters — no RecordCustomMetric call exists in the code. It is now repointed at the real Prometheus counter. THIS REQUIRES the New Relic Prometheus OpenMetrics integration (or nri-prometheus) to be scraping the api /metrics endpoint so the counter lands in NRDB as a dimensional Metric; if that integration is not configured this NR alert will not fire and the guaranteed-working equivalent is the 'StorageIamCreateFailures' rule in infra/k8s/prometheus-rules.yaml (instant-api group), which evaluates the same counter directly in Prometheus.",
   "enabled": true,
   "nrql": {
-    "query": "SELECT sum(newrelic.timeslice.value) FROM Metric WHERE metricTimesliceName = 'Custom/Storage/IamUsersFailed/create'"
+    "query": "SELECT sum(instant_storage_iam_users_failed_total) FROM Metric WHERE op = 'create'"
   },
   "terms": [
     {

newrelic/alerts/worker-stalled.json

@@ -1,10 +1,10 @@
 {
   "name": "instant-worker — no jobs processed in 10m",
   "type": "NRQL",
-  "description": "Page when the worker has processed zero jobs for 10 minutes. Catches stalled River pollers, deadlocks against the platform DB, and pod crash loops missed by k8s readiness probes. Source: Log records emitted by River job middleware in track 4.",
+  "description": "Page when the worker has processed zero jobs for 10 minutes. Catches stalled River pollers, deadlocks against the platform DB, and pod crash loops missed by k8s readiness probes. Source: Log records emitted by River job middleware (worker/internal/jobs/middleware.go) — message = 'jobs.middleware.work_ok' on every successfully processed job.",
   "enabled": true,
   "nrql": {
-    "query": "SELECT count(*) FROM Log WHERE service = 'worker' AND message = 'job.completed'"
+    "query": "SELECT count(*) FROM Log WHERE service = 'worker' AND message = 'jobs.middleware.work_ok'"
   },
   "terms": [
     {