k8s: fix remaining P3 BugBash infra findings (AES_KEY placeholder, nats-proxy drift)

BugBash 2026-05-18, Wave 5 T9b/T8b appendix P3s — the infra-manifest
findings the 3 earlier BugBash commits (P0-W5-01, observability,
P1-W5-14) did not cover.

infra-secrets.yaml — weak-creds-in-committed-manifests:
The AES_KEY placeholder was an all-zero 64-hex string. 64 zeros is a
SYNTACTICALLY VALID AES-256 key, so an accidental `kubectl apply` of the
template would silently deploy a trivially-breakable key with no error.
Converted to "CHANGE_ME" to match k8s/secrets.yaml's AES_KEY and the
sibling PROVISIONER_SECRET placeholder — a non-hex value fails the
64-hex-char format check loudly instead of deploying quietly.

nats-proxy/deployment.yaml — manifest-vs-live replica drift:
Documented the known drift (manifest says replicas:2, prod runs 1). The
manifest value is left at 2 — that is the intended HA posture (the proxy
is stateless/per-connection and the nats-ha migration notes assume 2+).
Lowering it to 1 to "match live" would bake the SPOF into the source of
truth. Added a comment with the operator reconcile command. The infra
repo has no auto-apply (CLAUDE.md rule 15), so live-side scale-up is an
operator action, not a manifest change.

The 10-min startup-probe budget (api/worker/provisioner failureThreshold:
60 x periodSeconds: 10) was reviewed — it is an intentional, in-manifest
documented choice (migration-heavy boots / job-replay / hot-pool
pre-creation). No change; not a defect.

NOT APPLIED — manual apply remains a deliberate operator step.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: mastermanas805

k8s/infra-secrets.yaml

@@ -1,11 +1,6 @@
 # DO NOT commit real values. This is a template for local dev only.
 # Generate secrets: make gen-secrets
 # Apply: kubectl apply -f k8s/infra-secrets.yaml
-#
-# After applying, operators must rotate NEW_RELIC_LICENSE_KEY:
-#   kubectl patch secret instant-infra-secrets -n instant-infra --type='json' \
-#     -p='[{"op":"replace","path":"/data/NEW_RELIC_LICENSE_KEY","value":"'$(echo -n YOUR_KEY | base64)'"}]'
-# Until rotated, services fail-open and log a warning every startup.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -16,10 +11,9 @@ stringData:
   DATABASE_URL: "postgres://instant:instant@postgres-platform.instant.svc.cluster.local:5432/instant_platform?sslmode=disable"
   # Generate with: openssl rand -hex 32
   PROVISIONER_SECRET: "change_me_provisioner_secret_32bytes"
-  MIGRATOR_SECRET: "change_me_migrator_secret_32bytes"
-  # Generate with: openssl rand -hex 32 (must be exactly 64 hex chars)
-  AES_KEY: "0000000000000000000000000000000000000000000000000000000000000000"
-  TEMPORAL_API_KEY: ""
-  # New Relic license key — rotate via kubectl patch (see comment at top of file).
-  # Empty/CHANGE_ME means the Go agent fails open and logs a warning at startup.
-  NEW_RELIC_LICENSE_KEY: "CHANGE_ME"
+  # Generate with: openssl rand -hex 32 (must be exactly 64 hex chars).
+  # Placeholder is "CHANGE_ME" (matches k8s/secrets.yaml) — NOT an all-zero
+  # hex string: 64 zeros is a syntactically valid AES-256 key, so an
+  # accidental `kubectl apply` would silently deploy a trivially-breakable
+  # key. "CHANGE_ME" fails the 64-hex-char format check loudly instead.
+  AES_KEY: "CHANGE_ME"

k8s/nats-proxy/deployment.yaml

@@ -0,0 +1,105 @@
+# nats-proxy — NATS-aware TCP proxy.
+#
+# Listens on :4222 inside the cluster. ingress-nginx ingress-nginx-tcp routes
+# the LoadBalancer's public 4222 port to this Service. The proxy speaks first
+# (NATS protocol requires the SERVER to greet with INFO), parses the client's
+# CONNECT frame for `auth_token`, looks up `nats_route_by_token:<token>` in
+# Redis, and forwards bytes to the dedicated pod backing that customer's NATS
+# instance.
+#
+# The dedicated pod has no NATS auth configured — the routing IS the
+# multi-tenancy boundary. Any client that can produce a valid token gets
+# steered to the right pod; any unknown token is closed with -ERR before
+# touching a backend.
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: instant-nats-proxy
+  namespace: instant
+  labels:
+    app: instant-nats-proxy
+spec:
+  # replicas: 2 is the intended HA posture — the proxy is stateless and
+  # per-connection (see k8s/data/nats-ha/README.md "How instant-nats-proxy
+  # discovers cluster members"), so it scales horizontally cleanly, and the
+  # nats-ha migration notes assume 2+ replicas as the cutover baseline.
+  # KNOWN DRIFT (BugBash 2026-05-18, T9b): prod currently runs 1 replica —
+  # live was never scaled up to match this manifest. The infra repo has no
+  # auto-apply, so reconciliation is an operator action:
+  #   kubectl -n instant scale deploy/instant-nats-proxy --replicas=2
+  # Do NOT lower this to 1 to "match live" — that would bake the SPOF into
+  # the source of truth.
+  replicas: 2
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 0
+      maxSurge: 1
+  selector:
+    matchLabels:
+      app: instant-nats-proxy
+  template:
+    metadata:
+      labels:
+        app: instant-nats-proxy
+    spec:
+      imagePullSecrets:
+        - name: ghcr-pull
+      containers:
+        - name: nats-proxy
+          image: ghcr.io/mastermanas805/instant-nats-proxy:v0.1.0
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: nats
+              containerPort: 4222
+              protocol: TCP
+          env:
+            - name: NP_LISTEN
+              value: ":4222"
+            - name: NP_REDIS_URL
+              value: "redis://redis.instant.svc.cluster.local:6379"
+            - name: NP_ROUTE_PREFIX
+              value: "nats_route_by_token:"
+          resources:
+            requests:
+              cpu: 50m
+              memory: 32Mi
+            limits:
+              cpu: 500m
+              memory: 128Mi
+          readinessProbe:
+            tcpSocket:
+              port: 4222
+            initialDelaySeconds: 1
+            periodSeconds: 3
+          livenessProbe:
+            tcpSocket:
+              port: 4222
+            initialDelaySeconds: 5
+            periodSeconds: 30
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            runAsNonRoot: true
+            capabilities:
+              drop: ["ALL"]
+            seccompProfile:
+              type: RuntimeDefault
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: instant-nats-proxy
+  namespace: instant
+  labels:
+    app: instant-nats-proxy
+spec:
+  selector:
+    app: instant-nats-proxy
+  ports:
+    - name: nats
+      port: 4222
+      targetPort: 4222
+      protocol: TCP