observability: alert + dashboard + catalog for instant_provisioner_drop_total (rule 25)#60
Merged
Conversation
…op_total (rule 25) Companion to provisioner PR #50 (customer-data drop chokepoint). The new server.guardedDrop emits instant_provisioner_drop_total{resource_type,backend, outcome} on every customer-data destruction the provisioner performs. - Prometheus group instant-provisioner-drop-audit: ProvisionerDropRateAbnormal (rate>0.2/s 10m — the truehomie burst signature, P1) + ProvisionerDropErrors (backend Deprovision failing, infra leaking, P2). - NR alerts: provisioner-drop-rate-abnormal.json, provisioner-drop-errors.json. - Dashboard tile: "Provisioner — customer-data drops by type/backend/outcome". - METRICS-CATALOG row. Infra has no auto-apply (rule 15) — operator applies prometheus-rules + imports NR alerts/dashboard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mastermanas805
added a commit
that referenced
this pull request
Jun 6, 2026
…cause vector) (#61) * sec(data): DORMANT postgres-customers admin lockdown (truehomie root-cause vector) Closes the OPEN root cause of the truehomie-db DROP incident (2026-06-03): a direct public-internet admin connection to the shared customer Postgres that can DROP DATABASE/ROLE with no audit_log row. CONFIRMED via config + SAFE checks (verify-don't-assert, 2026-06-06): - pg.instanode.dev -> 152.42.154.144 (same DO LB IP as api/redis/mongo); pg is publicly DNS-routed. - `nc -z pg.instanode.dev 5432` succeeds from the public internet (TCP handshake only -- NO auth, NO SQL attempted). - postgres-customers pod runs stock pgvector/pgvector:pg16 with NO custom pg_hba/postgresql.conf/POSTGRES_HOST_AUTH_METHOD and no config mount -> image default `host all all all scram-sha-256` (catch-all): the admin role (instant_cust) can authenticate from anywhere that reaches the listener. STILL HYPOTHESIS (not asserted): that an external actor actually used this to drop truehomie (we did not attempt auth); the instant-pg-proxy's own role-gate (its config lives in the separate InstaNode-dev/instant-pg-proxy repo). Fix (staged DORMANT -- operator-applies in a maintenance window; rule 15: infra has no auto-apply, high blast radius): - k8s/data/postgres-customers-lockdown.yaml: custom pg_hba.conf ConfigMap that REJECTS the admin role (instant_cust/postgres) from the public internet while still allowing in-cluster admin (provisioner/migrator/worker via pod CIDR) AND customer usr_* roles from anywhere (customer connect path preserved). - k8s/data/networkpolicy.yaml: truehomie context annotations + a DORMANT pg-proxy ingress rule (network-layer second line; admin boundary does not depend on it -- pg_hba role-reject holds regardless of SNAT source). - POSTGRES-CUSTOMERS-LOCKDOWN-RUNBOOK.md: confirmed-vs-hypothesis, legitimate- consumer map, pre-apply verification (live admin role name, pg-proxy SNAT/ NetPol interaction, hunt for public-admin automation), apply, legit-access- still-works verify, external-admin-closed verify (SAFE rejection test), and rollback. NOT applied to prod. Requires USER/OPERATOR review + maintenance window. Validated locally: yamllint (repo config) + kubeconform -strict (5/5 valid). Defense-in-depth: this is the infra half. App half already shipped (provisioner guardedDrop chokepoint + DDL-audit + CI guard, PR #50; drop-metric alert/tile/ catalog, infra PR #60). Audit: docs/ci/DATA-INTEGRITY-DROP-PATH-AUDIT.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * sec(data): fix admin lockdown for live findings — instanode_admin + proxy-IP reject Live pre-apply verification (do-nyc3-instant-prod, 2026-06-06) found the PR's pg_hba was incomplete and would NOT have closed the truehomie vector: 1. TWO superusers on the prod customer pod, not one: `instanode_admin` (the role api/worker connect with via CUSTOMER_DATABASE_URL — the CONFIRMED truehomie vector role) AND `instant_cust` (provisioner's POSTGRES_CUSTOMERS_URL POSTGRES_USER). The PR rejected only `instant_cust`, leaving the actual vector (`instanode_admin`) matching the catch-all customer allow. Both now rejected. 2. The pg-proxy SNAT problem: instant-pg-proxy is a normal in-cluster pod that re-originates TCP, so an external `-U instanode_admin` arrives SNAT'd to a proxy pod IP inside 10.0.0.0/8 — a plain `10.0.0.0/8 allow` would match it. Baseline probe confirmed the live vector is OPEN (external admin reaches scram). Added proxy-pod-IP `reject` lines (10.109.4.113, 10.109.0.101) ordered BEFORE the in-cluster allow so the SNAT'd external admin is rejected while legit consumer pods (direct svc DSN, different IPs) still authenticate. Documented the churn dependency + the durable pg-proxy role-gate follow-up. 3. Preserved live local/loopback/replication `trust` lines (readiness probe + backup pg_dumpall) instead of switching to peer. Added doadmin belt-suspenders. Runbook: added §3a (pg-proxy SNAT + proxy-IP churn operator note) and the L1-L5 live-confirmed findings table (supersedes H1-H3). NetworkPolicy NOT applied (verified not enforced in prod; applying as-is would default-deny the proxy path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rule-25 monitoring for the new
instant_provisioner_drop_totalmetric shipped in provisioner PR #50 (the customer-data drop chokepoint that hardens the OPEN truehomie-db DROP incident root-cause class).server.guardedDropemitsinstant_provisioner_drop_total{resource_type,backend,outcome}on every customer-data destruction the provisioner performs, alongside aevent=provisioner.dropaudit log line.Changes
k8s/prometheus-rules.yaml, new groupinstant-provisioner-drop-audit):ProvisionerDropRateAbnormal—rate(...{outcome="ok"}) > 0.2/sfor 10m. The truehomie burst signature (a spike of un-expected drops). P1.ProvisionerDropErrors—{outcome="error"} > 0for 30m, faceted byresource_type,backend(backend Deprovision failing → infra leaking). P2.provisioner-drop-rate-abnormal.json,provisioner-drop-errors.json.observability/METRICS-CATALOG.mdrow.Apply
Infra has no auto-apply (rule 15). Operator applies the PrometheusRule + imports the NR alerts/dashboard. The rate threshold (0.2/s) is an initial guard — tune against observed steady-state once the metric has baseline history.
Validation
prometheus-rules.yaml: valid YAML; new rules follow the existing group/rule structure.pool-reap-errors.jsonschema.🤖 Generated with Claude Code