infra(data-tier): S4 razorpay sig-fail monitoring + R6 nats PVC + R7 eviction protection + apply runbook

[mastermanas805]

What

Stateful instant-data hardening: durability (R6), eviction protection (R7), Razorpay inbound-webhook signature-failure monitoring (S4), and a consolidated operator apply runbook covering these PLUS the already-committed-but-unapplied S1/S2.

NO kubectl apply was performed. Per CLAUDE.md rule 15 this repo has no auto-apply — manifests ship as PR + runbook for the operator to apply in a maintenance window (real customer data is on these pods).

Changes

S4 — Razorpay webhook signature-failure monitoring (rule 25, mirrors the GitHub bad-signature alert)

Pairs the new instant_razorpay_webhook_sig_fail_total counter the api agent is adding. P2/abuse — the signature gate is the only thing between a forged subscription.charged and a free plan upgrade; the 400 short-circuits before dispatch so no plan_tier flips, this is visibility + WAF response.

• k8s/prometheus-rules.yaml — new instant-billing group, alert RazorpayWebhookSigFailSpike (increase(...[10m]) > 0, warning).
• newrelic/alerts/razorpay-webhook-sig-fail.json — P2 NRQL alert (same shape as github-webhook-bad-signature.json).
• newrelic/dashboards/instanode-reliability.json — billboard tile (row 84, threshold 1).
• observability/METRICS-CATALOG.md — catalog row.

R6 — NATS JetStream emptyDir{} → PVC

• k8s/data/nats.yaml — adds nats-jetstream-pvc (5Gi, cluster-default StorageClass = do-block-storage on DOKS), mounts it at /data/jetstream. Closes the # TODO: convert to PVC marker so queue stream/consumer state + messages survive pod restarts. Recreate strategy retained (RWO single-attach).

R7 — eviction protection under the 200%+ memory overcommit

• k8s/data/stateful-priority.yaml (new) — PriorityClass instant-data-critical (value 1000000) + one PodDisruptionBudget(minAvailable:1) each for postgres-customers / mongodb / redis-provision / nats. (minio excluded — retired 2026-05-20.)
• Right-sized resources.requests/limits on postgres-customers, mongodb, redis-provision — these were BestEffort QoS (no requests → first-evicted under pressure), now Burstable. nats already had requests.

Apply runbook

• k8s/DATA-TIER-APPLY-RUNBOOK.md (new) — operator apply-order (R7 → R6 → S1 → S2, least-risky-first) + per-step verification gates for R6/R7 AND the unapplied S1 (postgres-customers-lockdown.yaml) and S2 (networkpolicy.yaml). Includes:
  • the explicit S2 trap: the NetworkPolicy default-denies the in-cluster instant-pg-proxy unless the dormant allow-rule (lines ~88–103) is uncommented + matched to the live proxy ns/labels — applying as-is breaks every customer Postgres connection.
  • the load-bearing S1 verify: external psql -U instant_cust / -U instanode_admin -h pg.instanode.dev MUST be rejected after lockdown (safe connection-rejection probe, no SQL/DDL), while in-cluster admin + customer usr_* paths keep working.
• k8s/APPLY-CHECKLIST.md — cross-reference pointer to the new runbook.

Validation

• yamllint (CI relaxed config) — clean on all changed manifests.
• kubeconform -strict -ignore-missing-schemas -kubernetes-version 1.31.0 — full k8s/ sweep: 106 valid, 0 invalid, 0 errors (PrometheusRule CRD skipped, expected).
• JSON parse-checked: alert + dashboard.
• YAML structure-checked: instant-billing group nests under spec.groups; nats jetstream-data volume is persistentVolumeClaim, not emptyDir.

Operator apply order (in a maintenance window)

1. R7-A: kubectl apply -f k8s/data/stateful-priority.yaml
2. R7-B: apply mongodb.yaml + redis-provision.yaml, patch priorityClassName
3. R6: kubectl apply -f k8s/data/nats.yaml (+ priorityClassName patch; durability publish/restart test)
4. S1: apply lockdown ConfigMap + postgres-customers.yaml — verify external admin psql REJECTED
5. S2: edit the dormant pg-proxy rule first, then apply networkpolicy.yaml — verify customer path still works

🤖 Generated with Claude Code