feat(worker): e2e_cohort_sweep — purge stale is_test_cohort teams#92
Merged
Conversation
Belt-and-suspenders reaper for STALE synthetic test-cohort teams
(teams.is_test_cohort=true, api mig 067) leaked by a CI E2E run that
died before deleting its own ephemeral /internal/e2e/account (api #260).
- New River periodic job `e2e_cohort_sweep`, hourly, on the reconcile
queue, RunOnStart=true. Finds is_test_cohort teams older than a 2h TTL
(CI runs are minutes) not already terminal/destroying, capped at 25/run.
- Reuses the team-deletion executor's idempotent per-team teardown (the
same processTeam the orphan-sweep reconciler reuses): flip to
deletion_pending, deprovision resources + delete k8s namespaces + scrub
PII + tombstone. No second copy of the purge cascade.
- NEVER touches a non-cohort team: the candidate query filters on
is_test_cohort=true, AND each candidate is re-checked with isTestCohort
before purge, AND the deletion_pending UPDATE re-guards on the flag.
Inert in practice — no real team is is_test_cohort=true.
- Fail-safe: nil executor → WARN no-op; per-team failure isolated (stays
deletion_pending for next tick / orphan-sweep PASS 1); top-level query
error returns so River retries.
- Observability (rule 25): metric instant_e2e_cohort_swept_total{outcome}
(swept/failed/skipped_not_cohort) with NR alert PromQL documented inline;
INFO on work, DEBUG on idle tick.
- Tests: sqlmock unit tests for every outcome arm (nil-executor, query
error, scan error, idle tick, happy-path, recheck-not-cohort blocked,
recheck error, 0-row flip, flip error, executor error) + a DB-backed
integration test proving ONLY the stale cohort team is swept while a
fresh (within-TTL) cohort team AND a normal non-cohort team are left
fully intact. New testhelpers (SeedCohortTeamWithAge,
ClearPreexistingTestCohortTeams) carry in-package smoke coverage.
100% patch coverage on the new production code; make gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mastermanas805
enabled auto-merge (squash)
June 5, 2026 04:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new worker periodic job,
e2e_cohort_sweep, that purges STALE synthetic test-cohort teams as a belt-and-suspenders backstop. CI's E2E suite mints an ephemeral account per run via the api's/internal/e2e/accountendpoint (api #260), taggedteams.is_test_cohort=true(api mig 067), and is supposed to delete it at the end of the run. A CI run that crashes/times-out mid-flight leaks the account. This job reclaims it.How
cohortSweepInterval), reconcile queue,RunOnStart=true. CI runs are minutes, so hourly + a short TTL reclaims leaks the same hour.cohortSweepTTL = 2h— long enough to never reap an account a still-running CI job is using.is_test_cohort = true AND created_at < now() - 2h AND status NOT IN ('tombstoned','deletion_pending'), ordered oldest-first, capped at 25/run.processTeam(the same teardown the orphan-sweep reconciler reuses) — flip todeletion_pending, deprovision resources + delete k8s namespaces + scrub PII + tombstone. No second copy of the cascade.Safety — never touches a non-cohort team
The destructive path is guarded three times: the candidate
WHERE is_test_cohort=true, a per-candidateisTestCohort()re-check before purge (countedskipped_not_cohortif it disagrees), and a re-guard on the flag inside thedeletion_pendingUPDATE. Inert in practice — no real team isis_test_cohort=true(mig 067 defaults false). Fail-safe: nil executor → WARN no-op; per-team failure isolated (left for next tick / orphan-sweep PASS 1); top-level query error → River retry.Observability (rule 25)
instant_e2e_cohort_swept_total{outcome}(swept/failed/skipped_not_cohort), all label values primed at boot, NR alert PromQL documented inline on the metric. INFO on work, DEBUG on idle tick.Tests
SeedCohortTeamWithAge,ClearPreexistingTestCohortTeams) carry in-package smoke coverage (per the per-package coverage-attribution gotcha).100% patch coverage on the new production code;
make gategreen locally.🤖 Generated with Claude Code