sec(data): DORMANT postgres-customers admin lockdown (truehomie root-cause vector)

[mastermanas805]

What & why

Hardens the OPEN root cause of the truehomie-db DROP incident (2026-06-03) — a direct, public-internet admin connection to the shared customer Postgres that can DROP DATABASE/DROP ROLE with no audit_log row (the missing-audit-row signature of the incident). Matches docs/ci/DATA-INTEGRITY-DROP-PATH-AUDIT.md §truehomie root-cause hypotheses H1 (the confirmed vector).

DORMANT. NOT applied to prod. Requires USER/OPERATOR review + a maintenance window. Rule 15: infra has no auto-apply; this is high blast radius (shared customer-DB tier).

CONFIRMED via config + SAFE checks (verify-don't-assert, 2026-06-06)

• pg.instanode.dev → 152.42.154.144 — the same IP as api/redis/mongo.instanode.dev (the shared DO LoadBalancer fronting ingress-nginx). pg is publicly DNS-routed.
• nc -z -w5 pg.instanode.dev 5432 succeeds from the public internet — TCP handshake only (no auth, no SQL, no DDL attempted).
• The postgres-customers pod runs the stock pgvector/pgvector:pg16 image with NO custom pg_hba.conf / postgresql.conf / POSTGRES_HOST_AUTH_METHOD and no config mount → the image default host all all all scram-sha-256 (catch-all) is in effect: the admin role (instant_cust) can authenticate from any source that reaches the listener.
• postgres-customers Service is ClusterIP; the public exposure is via the external instant-pg-proxy + ingress-nginx tcp-services (not in this repo).
• A postgres-customers-ingress NetworkPolicy already exists (provisioner/migrator/worker only); its prod-apply state is unverified (no auto-apply).

Still hypothesis (NOT asserted)

• That an external actor actually authenticated as admin over this path (we did not attempt auth — out of scope). C1–C3 prove the path is open, not that it was used.
• The instant-pg-proxy's own role-gate — its config lives in the separate InstaNode-dev/instant-pg-proxy repo.

Net: the hardening agent's #1 hypothesis is corroborated, not refuted.

What this changes

• k8s/data/postgres-customers-lockdown.yaml (new) — custom pg_hba.conf ConfigMap that rejects the admin role (instant_cust/postgres) from the public internet while still allowing in-cluster admin (provisioner/migrator/worker via pod CIDR 10.0.0.0/8) and customer usr_* roles from anywhere (customer connect path preserved). Keyed on the role name so it holds even if pg-proxy SNATs customer traffic to a pod IP.
• k8s/data/networkpolicy.yaml (edit) — truehomie-context annotations + a dormant (commented) pg-proxy ingress rule (network-layer second line; the admin boundary does NOT depend on it).
• POSTGRES-CUSTOMERS-LOCKDOWN-RUNBOOK.md (new) — confirmed-vs-hypothesis, the legitimate-consumer map, pre-apply verification (confirm the live admin role name; the pg-proxy SNAT/NetPol interaction; hunt for any public-admin automation), apply (apply CM → patch pod to mount + -c hba_file=... → rollout), legit-access-still-works verify (provisioner admin select 1, real /db/new + customer connect, backup CronJob smoke), external-admin-closed verify (SAFE rejection test — no SQL, no valid creds needed), and rollback.

Legitimate consumers preserved

provisioner (admin DSN, in-cluster), migrator, worker (read), customers (usr_* via pg-proxy), backup CronJob — all preserved (see runbook §2). The one thing closed: a psql -U instant_cust from outside the cluster.

Safety

• Nothing applies automatically. The pg_hba change is online-reloadable; the pod patch is a brief single-replica restart (provisioner retries, customers reconnect).
• Biggest break-risk = an undocumented consumer using public admin access; the runbook's pre-apply step (e) hunts for exactly that. If found → migrate it in-cluster before apply.
• Validated locally: yamllint (repo config) + kubeconform -strict (5/5 valid).

Defense-in-depth context

Infra half of the truehomie fix. App half already shipped: provisioner guardedDrop chokepoint + DDL-audit + CI guard (PR #50); drop-metric alert/tile/catalog (infra PR #60).

🤖 Generated with Claude Code