Skip to content

feat(worker): e2e_cohort_sweep — purge stale is_test_cohort teams#92

Merged
mastermanas805 merged 1 commit into
masterfrom
feat/e2e-cohort-sweep
Jun 5, 2026
Merged

feat(worker): e2e_cohort_sweep — purge stale is_test_cohort teams#92
mastermanas805 merged 1 commit into
masterfrom
feat/e2e-cohort-sweep

Conversation

@mastermanas805

Copy link
Copy Markdown
Member

What

A new worker periodic job, e2e_cohort_sweep, that purges STALE synthetic test-cohort teams as a belt-and-suspenders backstop. CI's E2E suite mints an ephemeral account per run via the api's /internal/e2e/account endpoint (api #260), tagged teams.is_test_cohort=true (api mig 067), and is supposed to delete it at the end of the run. A CI run that crashes/times-out mid-flight leaks the account. This job reclaims it.

How

  • Schedule: hourly (cohortSweepInterval), reconcile queue, RunOnStart=true. CI runs are minutes, so hourly + a short TTL reclaims leaks the same hour.
  • TTL: cohortSweepTTL = 2h — long enough to never reap an account a still-running CI job is using.
  • Candidate query: is_test_cohort = true AND created_at < now() - 2h AND status NOT IN ('tombstoned','deletion_pending'), ordered oldest-first, capped at 25/run.
  • Purge mechanism: reuses the team-deletion executor's idempotent processTeam (the same teardown the orphan-sweep reconciler reuses) — flip to deletion_pending, deprovision resources + delete k8s namespaces + scrub PII + tombstone. No second copy of the cascade.

Safety — never touches a non-cohort team

The destructive path is guarded three times: the candidate WHERE is_test_cohort=true, a per-candidate isTestCohort() re-check before purge (counted skipped_not_cohort if it disagrees), and a re-guard on the flag inside the deletion_pending UPDATE. Inert in practice — no real team is is_test_cohort=true (mig 067 defaults false). Fail-safe: nil executor → WARN no-op; per-team failure isolated (left for next tick / orphan-sweep PASS 1); top-level query error → River retry.

Observability (rule 25)

instant_e2e_cohort_swept_total{outcome} (swept/failed/skipped_not_cohort), all label values primed at boot, NR alert PromQL documented inline on the metric. INFO on work, DEBUG on idle tick.

Tests

  • Unit (sqlmock): every outcome arm — nil-executor no-op, candidate-query error, row-scan error, idle tick, happy-path swept, recheck-not-cohort blocked, recheck error skip, 0-row flip skip, flip error, executor error.
  • Integration (real Postgres): seeds a STALE cohort team + a FRESH (within-TTL) cohort team + a NORMAL non-cohort team, runs the job, asserts only the stale cohort team is tombstoned/scrubbed while the other two are fully intact. Plus a nil-executor no-op.
  • New testhelpers (SeedCohortTeamWithAge, ClearPreexistingTestCohortTeams) carry in-package smoke coverage (per the per-package coverage-attribution gotcha).

100% patch coverage on the new production code; make gate green locally.

🤖 Generated with Claude Code

Belt-and-suspenders reaper for STALE synthetic test-cohort teams
(teams.is_test_cohort=true, api mig 067) leaked by a CI E2E run that
died before deleting its own ephemeral /internal/e2e/account (api #260).

- New River periodic job `e2e_cohort_sweep`, hourly, on the reconcile
  queue, RunOnStart=true. Finds is_test_cohort teams older than a 2h TTL
  (CI runs are minutes) not already terminal/destroying, capped at 25/run.
- Reuses the team-deletion executor's idempotent per-team teardown (the
  same processTeam the orphan-sweep reconciler reuses): flip to
  deletion_pending, deprovision resources + delete k8s namespaces + scrub
  PII + tombstone. No second copy of the purge cascade.
- NEVER touches a non-cohort team: the candidate query filters on
  is_test_cohort=true, AND each candidate is re-checked with isTestCohort
  before purge, AND the deletion_pending UPDATE re-guards on the flag.
  Inert in practice — no real team is is_test_cohort=true.
- Fail-safe: nil executor → WARN no-op; per-team failure isolated (stays
  deletion_pending for next tick / orphan-sweep PASS 1); top-level query
  error returns so River retries.
- Observability (rule 25): metric instant_e2e_cohort_swept_total{outcome}
  (swept/failed/skipped_not_cohort) with NR alert PromQL documented inline;
  INFO on work, DEBUG on idle tick.
- Tests: sqlmock unit tests for every outcome arm (nil-executor, query
  error, scan error, idle tick, happy-path, recheck-not-cohort blocked,
  recheck error, 0-row flip, flip error, executor error) + a DB-backed
  integration test proving ONLY the stale cohort team is swept while a
  fresh (within-TTL) cohort team AND a normal non-cohort team are left
  fully intact. New testhelpers (SeedCohortTeamWithAge,
  ClearPreexistingTestCohortTeams) carry in-package smoke coverage.

100% patch coverage on the new production code; make gate green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mastermanas805
mastermanas805 enabled auto-merge (squash) June 5, 2026 04:56
@mastermanas805
mastermanas805 merged commit 53078dd into master Jun 5, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant