feat(worker): audit-only orphan-customer-DB / redis-namespace sweep (flag-gated OFF) (#102)

* feat(worker): audit-only orphan-customer-DB / redis-namespace sweep (flag-gated OFF)

New River periodic job orphan_db_sweep (hourly, reconcile queue, UniqueOpts)
that addresses the ~25 orphaned customer DB / redis namespace drain-backlog —
in DETECTION / DRY-RUN mode only. It lists instant-customer-* namespaces, flags
the ones whose token has NO non-terminal (pending/active/paused/suspended)
resources row and is past the provisioning grace window, then LOGS each
candidate (token masked via logsafe.Token) and emits the candidate metrics.
It DROPS NOTHING in audit-only mode.

truehomie-2026-06-03 safety: there is NO manual / raw DROP anywhere in this
job. The destructive teardown sits behind a SECOND flag and, when (and only
when) enabled, routes through the AUDITED provisioner DeprovisionResource
chokepoint — the same path the TTL reaper (expire.go) uses. For this PR that
path is intentionally unreachable-by-default.

Two flag gates, BOTH default OFF / fail-closed:
  ORPHAN_DB_SWEEP_ENABLED            — master flag; off → Work is a DEBUG no-op
                                       (no namespace List, no DB read, no metric).
  ORPHAN_DB_SWEEP_DESTRUCTIVE_ENABLED — destructive flag; meaningless unless the
                                       master is also on AND a provisioner is
                                       wired. Routes through the audited
                                       chokepoint only.

Fail-safe / fail-open: a namespace-List error or a live-token DB-read error
degrades to ZERO candidates (never an empty-set that a destructive caller could
read as "drop everything"); a candidate whose token reappears live at the
destructive re-confirm is SKIPPED; a generic (unmapped-kind) orphan is SKIPPED
(no proven backing type → no guessed DROP). When in doubt, skip + log.

Metrics (lazy *Vec, both labels primed in metrics_test):
  instant_orphan_db_sweep_candidates_total{kind}   — counter, kind in
    {customer_namespace, redis_namespace}
  instant_orphan_db_sweep_candidates_current{kind} — gauge, current backlog
Alert + dashboard + catalog live in the infra repo (not owned here) — see PR
body for the exact metric names + suggested alerts (rule 25 follow-up).

Tests: candidate detection (orphan vs live vs pending vs within-grace), kind
classification, both flag gates (off → no-op), masking, fail-open paths, and
that the destructive deprovisioner is NEVER called in audit-only mode. New file
at 100% statement coverage. make gate green (build + vet + go test -short).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(worker): 100% patch coverage — extract typed-nil-safe deprovisioner helper

diff-cover flagged the `if provClient != nil` branch in the StartWorkers wiring
(integration-only, not unit-reachable). Extract the typed-nil-safe conversion
into orphanDBSweepDeprovisionerFor (mirrors NewExpireAnonymousWorker's handling)
and unit-test both arms directly, so the wiring call site is a single
non-branching expression covered by TestStartWorkers_FullBoot and the branch
logic is covered by the new unit test. New code back to 100% patch coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: mastermanas805

internal/config/config.go

@@ -206,6 +206,31 @@ type Config struct {
 	// pathological flapping). Enabling is an operator action after a canary.
 	DeployScaleToZeroEnabled     bool // DEPLOY_SCALE_TO_ZERO_ENABLED — master flag (default false)
 	DeployScaleToZeroIdleMinutes int  // DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES — idle threshold (default 30)
+
+	// Audit-only orphan-customer-DB / orphan-redis-namespace sweep
+	// (orphan_db_sweep.go). TWO independent flags, both default OFF / fail-closed
+	// (project_feature_flag_decision: every new feature ships flag-gated,
+	// default-off, fail-closed).
+	//
+	// OrphanDBSweepEnabled — the MASTER flag. When false (the default) the sweep
+	// Work method no-ops immediately: no namespace List, no DB read, no metric,
+	// no log beyond a single DEBUG. A single env flip turns the whole detection
+	// layer on. In audit-only mode (the destructive flag below OFF) an enabled
+	// sweep ONLY logs masked orphan candidates + emits the candidate metrics — it
+	// drops NOTHING.
+	//
+	// OrphanDBSweepDestructiveEnabled — the SECOND, destructive flag. Default
+	// OFF. It is meaningless unless the master flag is ALSO on. When (and only
+	// when) BOTH are true does the sweep route a confirmed orphan through the
+	// AUDITED provisioner DeprovisionResource chokepoint (the SAME path the TTL
+	// reaper uses) — NEVER a manual/raw DROP. This is the truehomie-2026-06-03
+	// safety posture: an active Pro customer's DB+role were dropped by an
+	// unaudited path, so this job must never improvise a DROP. For THIS PR the
+	// destructive path is wired but intentionally left UNREACHABLE-BY-DEFAULT:
+	// we ship audit-only, review the dry-run candidate list, and only then (in a
+	// later, deliberate operator action) consider lighting the destructive flag.
+	OrphanDBSweepEnabled            bool // ORPHAN_DB_SWEEP_ENABLED — master flag (default false)
+	OrphanDBSweepDestructiveEnabled bool // ORPHAN_DB_SWEEP_DESTRUCTIVE_ENABLED — destructive flag (default false; requires master on; routes through audited provisioner only)
 }
 
 // ErrMissingConfig is returned when a required env var is absent.
@@ -340,6 +365,13 @@ func Load() *Config {
 		// Scale-to-zero idle-scaler (Task #54). Default OFF; idle threshold
 		// default 30 min (parsed below).
 		DeployScaleToZeroEnabled: os.Getenv("DEPLOY_SCALE_TO_ZERO_ENABLED") == "true",
+
+		// Audit-only orphan-DB sweep. BOTH default OFF / fail-closed. The
+		// destructive flag is inert unless the master flag is also on (the sweep
+		// enforces that ordering at runtime). Shipping audit-only: the
+		// destructive flag stays unset until we've reviewed the dry-run list.
+		OrphanDBSweepEnabled:            os.Getenv("ORPHAN_DB_SWEEP_ENABLED") == "true",
+		OrphanDBSweepDestructiveEnabled: os.Getenv("ORPHAN_DB_SWEEP_DESTRUCTIVE_ENABLED") == "true",
 	}
 
 	// DEPLOY_SCALE_TO_ZERO_IDLE_MINUTES: minutes of no-activity before an app is

internal/config/config_test.go

@@ -124,6 +124,44 @@ func TestLoad_Defaults(t *testing.T) {
 	if cfg.SESTemplateNames == nil || len(cfg.SESTemplateNames) != 0 {
 		t.Errorf("SESTemplateNames = %v", cfg.SESTemplateNames)
 	}
+	// Audit-only orphan-DB sweep flags both default OFF / fail-closed.
+	if cfg.OrphanDBSweepEnabled {
+		t.Error("OrphanDBSweepEnabled should default false (fail-closed)")
+	}
+	if cfg.OrphanDBSweepDestructiveEnabled {
+		t.Error("OrphanDBSweepDestructiveEnabled should default false (fail-closed)")
+	}
+}
+
+// TestLoad_OrphanDBSweepFlags pins the two env-driven flags of the audit-only
+// orphan-DB sweep: each is set ONLY by its exact env var being literally "true".
+func TestLoad_OrphanDBSweepFlags(t *testing.T) {
+	t.Run("both true", func(t *testing.T) {
+		clearEnv(t)
+		t.Setenv("DATABASE_URL", "postgres://localhost/db")
+		t.Setenv("ORPHAN_DB_SWEEP_ENABLED", "true")
+		t.Setenv("ORPHAN_DB_SWEEP_DESTRUCTIVE_ENABLED", "true")
+		cfg := Load()
+		if !cfg.OrphanDBSweepEnabled {
+			t.Error("OrphanDBSweepEnabled should be true when env is 'true'")
+		}
+		if !cfg.OrphanDBSweepDestructiveEnabled {
+			t.Error("OrphanDBSweepDestructiveEnabled should be true when env is 'true'")
+		}
+	})
+	t.Run("non-true is off", func(t *testing.T) {
+		clearEnv(t)
+		t.Setenv("DATABASE_URL", "postgres://localhost/db")
+		t.Setenv("ORPHAN_DB_SWEEP_ENABLED", "1")     // not exactly "true"
+		t.Setenv("ORPHAN_DB_SWEEP_DESTRUCTIVE_ENABLED", "yes")
+		cfg := Load()
+		if cfg.OrphanDBSweepEnabled {
+			t.Error("OrphanDBSweepEnabled should be false for non-'true' value")
+		}
+		if cfg.OrphanDBSweepDestructiveEnabled {
+			t.Error("OrphanDBSweepDestructiveEnabled should be false for non-'true' value")
+		}
+	})
 }
 
 // TestLoad_DeployScaleToZeroIdleMinutes exercises the env-parse branch for