feat: per-company screenshot retention policy + nightly cleanup cron#157
Merged
Conversation
Adds an opt-in retention policy on the time-tracker screenshot pipeline.
Visible on /settings/setting to the company owner (roleType === 1)
only. When enabled, a nightly cron permanently deletes trackshots
older than the configured window (3 / 6 / 12 / 24 months) from both
the per-tenant TimeSheet.trackShots subdoc array and from Wasabi
(main object + every thumbnail variant).
Backend
=======
- utils/mongo-handler/schema.js: new `screenshotRetention` Map field
on the global companies schema. Stores `enabled`, `maxAgeMonths`,
`enabledAt`, `enabledBy`, `lastRunAt`, `lastRunStats`,
`firstRunCompletedAt`, `runningSince`. Backward-compatible — legacy
companies without the field read as `enabled: false`.
- Modules/ScreenshotRetention/: new module.
helper.js — policy read/write, preview counter, per-company
cleanup workflow, and the cron entry point.
Production-ready guarantees:
* Wasabi delete BEFORE db $pull so transient
Wasabi failures leave the db record intact and
the next nightly run retries (no permanent
orphans).
* Per-trackshot main + 4 thumbnail keys deleted
(sizes hard-coded from thumbnail.json).
* Filters by trackshot.screenShotTime (epoch ms),
not parent TimeSheet timestamp.
* Advisory `runningSince` lock prevents
double-runs; stale locks (>4h) are reclaimed.
* First-run safety cap (50k deletions) for the
initial cleanup on legacy data; lifted once
`firstRunCompletedAt` is stamped.
* Bounded company concurrency (5 in parallel)
via Promise.allSettled.
controller.js — three endpoints with owner role check:
GET /api/v1/screenshot-retention
GET /api/v1/screenshot-retention/preview
PUT /api/v1/screenshot-retention
Owner check looks up the per-tenant
`company_users` doc for the caller and confirms
`roleType === 1`. Returns 403 on mismatch.
routes.js — endpoint registration.
init.js — module bootstrap (matches existing convention).
- index.js: register the new module beside the rest of
`initializeControllers()`.
- cron.js: removed the broken `cleanUpTrackshot()` call (was
referencing an unimported binding and never executed). Added a new
schedule at 00:30 UTC that invokes
`screenshotRetention.runRetentionForAllCompanies()`. Off-peak vs the
other midnight jobs so a heavy cleanup doesn't compound with the
bucket-size + AI-reset jobs on the same minute.
Frontend
========
- frontend/src/components/molecules/Setting/SettingScreenshotRetention.vue:
new card mounted on /settings/setting. Renders only for
`companyUser.roleType === 1`. Toggle + retention-window dropdown +
last-run telemetry. Enabling fires a SweetAlert confirmation that
shows the preview-count from the new GET preview endpoint so the
owner knows exactly what will be deleted on the next nightly run.
- frontend/src/views/Settings/Setting/Setting.vue: mount the new
component in the existing right-hand column. Self-hides for
non-owners.
- frontend/src/locales/en.js: new ScreenshotRetention.* keys (heading,
toggle/window labels, confirmation copy, last-run telemetry). Other
locales fall back to English via vue-i18n.
Out of scope (deferred)
=======================
- Translation backfill for non-English locales.
- A `RetentionAuditLog` collection for per-run history beyond
`lastRunStats`. The cron emits structured logs in the meantime.
- A "dry run" mode that lets the owner see what would be deleted
without enabling the policy. The preview endpoint already gives the
count; a per-record preview would need its own UI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…Retention component
Independent review of the original commit surfaced three blockers and
several correctness issues. This is a follow-up commit on the same
branch that addresses all of them.
BLOCKERS
========
1. **Routes had no JWT middleware.** Config/setMiddleware.js gates auth
via path allowlists. The new `/api/v1/screenshot-retention` and
`/api/v1/screenshot-retention/preview` paths were NOT listed in
either `verifyJWTTokenWithCRoute` or `verifyJWTToken`, so the
endpoints shipped completely unauthenticated — anyone on the
internet could toggle retention for any company. Added both paths
to `verifyJWTTokenWithCRoute` so the JWT middleware fires and
populates `req.uid` / `req.aud`.
2. **Thumbnail keys were generated in the wrong dimension order.** The
upload-time call at Modules/storage/wasabi/controller.js:298 passes
`(thu.height, thu.width)` into a function whose params are
`(width, height)`, so the stored filename is effectively
`<base>-<height>x<width>.<ext>`. The cleanup helper was building
`<base>-<width>x<height>.<ext>` and missing every thumbnail variant
on every retention run — leaving four orphan Wasabi objects per
trackshot, permanently. derivThumbnailKeys now mirrors the
upload-time swap with a comment explaining the pre-existing bug
in the upload path (out of scope to fix here).
3. **Cursor-based pagination replaces skip-based scan.** The previous
loop advanced `pagedSkip += docs.length` after each batch, but
the `$pull` inside the loop removes docs from the match set —
meaning the next page's skip lands past where it should. Most of
the DB was being missed on legacy-data tenants. Replaced with
`_id > lastSeenId` cursor that's immune to in-flight match-set
shrinkage.
SECURITY
========
4. **Controller now reads userId from `req.uid`** (set by
Config/jwt.js#checkToken from the verified token's `uid` claim),
not from `req.body.userId`. The old code let a non-owner pass the
owner's userId in the body and pass the role check. Status codes
updated: missing userId → 401 (authentication required) rather
than 400.
HIGH
====
5. **Legacy string-typed `screenShotTime` is now handled.** Multipart
form uploads coerce numbers to strings at write time, so the
`screenShotTime` field on legacy trackshots is stored as a string.
The previous `$lt: cutoffMs` (number) didn't match strings.
- `countOldTrackshots` now uses `$convert` with
`to: 'long', onError: null` inside the $filter so the comparison
works regardless of the field's actual storage type.
- `runRetentionForCompany`'s loose DB query + strict in-memory
filter coerces via `Number(t.screenShotTime)` and skips
non-finite values.
6. **`firstRunCompletedAt` is now stamped only when a real cleanup
completed.** Previous condition `deletedCount < cap` stamped the
marker on a zero-deletion run (e.g. a new company with no eligible
data), which lifted the first-run safety cap before any real bulk
data accumulated. New condition: scan exhausted (no cap hit, no
error) AND deletedCount > 0.
7. **Empty-image trackshots no longer inflate `deletedCount`.** The
scan loop now pre-filters them out (`oldShots` requires `t.image`),
and `deleteTrackshotObjects` returns `skipped: true` rather than
`mainDeleted: true` if it's ever called with a missing key.
Stats now track `skippedCount` separately.
MEDIUM
======
8. **Removed the no-op `updateCompanyPolicy(companyId, {}, {})`
call** that hit the empty-patch early-return at line 176 without
writing anything. Lock acquisition is a single `stampMasterField`
call, which is at least document-atomic. Cluster-aware leader
election is still out of scope (documented).
9. **`lastRunStats` now records partial-run errors and cap-hit state.**
New stats fields: `skippedCount`, `hitCap`, `error`. Ops can tell
from the master doc whether the run completed cleanly, hit the
cap, or threw mid-loop.
10. **Frontend now uses `watch(isOwner, …, {immediate: true})`** instead
of one-shot `onMounted`. The Vuex `companyUserDetail` store
sometimes hydrates after the component mounts on soft route
changes; the one-shot mount would never call `loadPolicy()` in
that case and the card stayed on hard-coded defaults forever.
LOW
===
11. **`extractKey` regex tightened** from `[a-f0-9]{20,}` to
`[a-f0-9]{24}` (exact ObjectId length) and now decodes URL
path segments so percent-encoded keys map back to their
canonical storage form.
12. **`getS3Client` doc + body cleaned up.** `s3Client` is not exported
from `wasabi/controller.js`, so the old "fall back if not
available" branch was always taken. Removed the misleading
primary path; we just build the client from `awsRef` and
memoise it.
Test plan addendum
==================
- Hit `GET /api/v1/screenshot-retention` without a JWT → 401.
- Hit `PUT /api/v1/screenshot-retention` with a forged body
`userId` of the owner from a non-owner JWT → 403 (controller
uses `req.uid`, not body).
- On a tenant with legacy string-typed `screenShotTime`:
- GET /preview returns a non-zero count for the configured window.
- A nightly run deletes those records + their (height×width-named)
thumbnail variants.
- Confirm thumbnail keys deleted in Wasabi by listing the bucket
after a run — there should be no orphans for the deleted main
keys.
- Confirm pagination handles a tenant whose every TimeSheet has at
least one expired trackshot (all docs drop from match set on
first $pull): the cursor variant should still terminate cleanly
and process every doc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in retention policy on the time-tracker screenshot pipeline. Visible on /settings/setting to the company owner (roleType === 1) only. When enabled, a nightly cron permanently deletes trackshots older than the configured window (3 / 6 / 12 / 24 months) from both the per-tenant TimeSheet.trackShots subdoc array and from Wasabi (main object + every thumbnail variant).
Backend
utils/mongo-handler/schema.js: new
screenshotRetentionMap field on the global companies schema. Storesenabled,maxAgeMonths,enabledAt,enabledBy,lastRunAt,lastRunStats,firstRunCompletedAt,runningSince. Backward-compatible — legacy companies without the field read asenabled: false.Modules/ScreenshotRetention/: new module.
helper.js — policy read/write, preview counter, per-company
cleanup workflow, and the cron entry point.
Production-ready guarantees:
* Wasabi delete BEFORE db $pull so transient
Wasabi failures leave the db record intact and
the next nightly run retries (no permanent
orphans).
* Per-trackshot main + 4 thumbnail keys deleted
(sizes hard-coded from thumbnail.json).
* Filters by trackshot.screenShotTime (epoch ms),
not parent TimeSheet timestamp.
* Advisory
runningSincelock preventsdouble-runs; stale locks (>4h) are reclaimed.
* First-run safety cap (50k deletions) for the
initial cleanup on legacy data; lifted once
firstRunCompletedAtis stamped.* Bounded company concurrency (5 in parallel)
via Promise.allSettled.
controller.js — three endpoints with owner role check:
GET /api/v1/screenshot-retention
GET /api/v1/screenshot-retention/preview
PUT /api/v1/screenshot-retention
Owner check looks up the per-tenant
company_usersdoc for the caller and confirmsroleType === 1. Returns 403 on mismatch.routes.js — endpoint registration.
init.js — module bootstrap (matches existing convention).
index.js: register the new module beside the rest of
initializeControllers().cron.js: removed the broken
cleanUpTrackshot()call (was referencing an unimported binding and never executed). Added a new schedule at 00:30 UTC that invokesscreenshotRetention.runRetentionForAllCompanies(). Off-peak vs the other midnight jobs so a heavy cleanup doesn't compound with the bucket-size + AI-reset jobs on the same minute.Frontend
frontend/src/components/molecules/Setting/SettingScreenshotRetention.vue: new card mounted on /settings/setting. Renders only for
companyUser.roleType === 1. Toggle + retention-window dropdown + last-run telemetry. Enabling fires a SweetAlert confirmation that shows the preview-count from the new GET preview endpoint so the owner knows exactly what will be deleted on the next nightly run.frontend/src/views/Settings/Setting/Setting.vue: mount the new component in the existing right-hand column. Self-hides for non-owners.
frontend/src/locales/en.js: new ScreenshotRetention.* keys (heading, toggle/window labels, confirmation copy, last-run telemetry). Other locales fall back to English via vue-i18n.
Out of scope (deferred)
RetentionAuditLogcollection for per-run history beyondlastRunStats. The cron emits structured logs in the meantime.Pull Request Template Chooser
Please click the link that matches your contribution type to load the correct format.
Bug Fix
Use this for fixing broken logic or UI glitches.
New Feature
Use this for adding new functionality or components.
Refactor
Use this for code cleanup, performance tweaks, or technical debt.
General Summary
If you don't want to use a specific template, please provide a brief summary of your changes below.