Skip to content

fix(engine): add retention cron to delete expired UserSession rows#4216

Closed
AlexlaGuardia wants to merge 1 commit into
hatchet-dev:mainfrom
AlexlaGuardia:fix/3913-usersession-retention-cron
Closed

fix(engine): add retention cron to delete expired UserSession rows#4216
AlexlaGuardia wants to merge 1 commit into
hatchet-dev:mainfrom
AlexlaGuardia:fix/3913-usersession-retention-cron

Conversation

@AlexlaGuardia

Copy link
Copy Markdown

Fixes #3913.

Problem

The UserSession table grows unboundedly. On the reporter's self-hosted deployment it reached 2.6M rows (~1 GB) in 30 days, all userId = NULL, written once and never read again. PR #3923 fixed the root-cause insert (Bearer-authed requests no longer create phantom sessions), but nothing purges the rows that already expired: the only DELETE on the table is DeleteUserSession by id (explicit logout), and tenant retention doesn't cover this table.

This is the follow-up @gregfurman asked for in #3913 — "we should also be adding a cron to periodically clean up these expired entries."

Fix

A periodic cleanup in the retention controller, modelled on the existing runCleanupOldWorkers / runDeleteMessageQueueItems jobs:

  • CleanupExpiredUserSessions sqlc queryDELETE of expired rows via a LIMITed id IN (SELECT …) subquery, so each call deletes a bounded batch.
  • UserSessionRepository.DeleteExpired(ctx, batchSize) — runs one batch inside a statement-timeout-bounded transaction (PrepareTxWithStatementTimeout, 3 min), returning whether a full batch was deleted. Same shape as workerRepository.CleanupOldWorkers.
  • runCleanupExpiredUserSessions retention job — loops DeleteExpired until a partial batch (backlog drained), under a 30-min context timeout. Registered as a 24h gocron job with SingletonMode, mirroring the worker-retention block.
  • Index migration (v1_0_118) — CREATE INDEX CONCURRENTLY on "UserSession"("expiresAt"). Without it each batch sequential-scans the table, which matters precisely because the backlog can be in the millions. Added CONCURRENTLY + -- +goose no transaction per the existing precedent (v1_0_47).

Notes / decisions

  • Always-on, no config flag. I followed the queueRetention / dataRetention precedent (always-on global cleanups) rather than workerRetention (which has an EnableWorkerRetention flag only because it defaults off). Deleting expired sessions is universally safe — they're unusable by definition — so there's no operator reason to disable it. A WithSessionRetention opt exists if you'd prefer it gated behind config; say the word and I'll wire EnableSessionRetention through server.go / loader.go / run.go to match the worker knob.
  • Batch size 10000, same as CleanupOldWorkers.

Testing

go build + go vet clean on the changed packages. The retention jobs are integration-level (no existing unit tests for retention/ or the session repo), so I didn't add unit tests to match the existing convention — happy to add an integration test that seeds expired rows and asserts the drain if you'd like one before merge.

AI usage disclosure

Per CONTRIBUTING.md#ai-usage: implemented with Claude Code (Opus). It traced the issue through the retention controller and repository layers, matched the existing CleanupOldWorkers cleanup pattern, regenerated the sqlc binding, and added the index migration. I reviewed every change against the repo's conventions before opening. Same human-directed workflow and disclosure as #4180 and #4214.

The UserSession table grows unboundedly: PR hatchet-dev#3923 stopped Bearer-authed
requests from creating phantom sessions, but nothing purges rows that have
already expired. The only DELETE is by id (explicit logout), and tenant
retention does not cover this table.

Add a periodic cleanup to the retention controller, modelled on the existing
runCleanupOldWorkers / runDeleteMessageQueueItems jobs:

- CleanupExpiredUserSessions sqlc query: batched DELETE of expired rows
- UserSessionRepository.DeleteExpired: one batch per call inside a
  statement-timeout-bounded transaction, returning whether more remain
- runCleanupExpiredUserSessions: 24h gocron job (SingletonMode) that drains
  the backlog under a 30-minute context timeout
- index migration on UserSession(expiresAt) so each batch does not seq-scan

Fixes hatchet-dev#3913
@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown

@AlexlaGuardia is attempting to deploy a commit to the Hatchet Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added the engine Related to the core Hatchet engine label Jun 16, 2026
@gregfurman

Copy link
Copy Markdown
Collaborator

Hey @AlexlaGuardia. Thanks for the contribution here but unfortunately this feature is already covered by #4105

@AlexlaGuardia

Copy link
Copy Markdown
Author

Makes sense, thanks for checking. #4105 covers it, closing this out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine Related to the core Hatchet engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UserSession table in Hatchet Postgres grows unboundedly from Bearer API requests

2 participants