feat(platform): ADR-0057 system data lifecycle + P0 telemetry-growth stopgap#2079
Merged
Conversation
…stopgap A 20s scheduled digest flow in app-showcase grew dev.db to 260MB+ over a multi-day `pnpm dev` — all platform-generated telemetry, zero business data. Root causes: (1) no retention contract on platform objects, (2) audit+activity dual-write amplifying every internal mutation (~76% of row growth), (3) SQLite auto_vacuum=NONE so the file never shrinks. ADR-0057 makes data lifecycle a declarable, runtime-enforced metadata concern (classify → declare → enforce via LifecycleService → reclaim → gate), building on ADR-0052's bounded contexts. This commit ships the P0 stopgap: - plugin-audit: exclude operational/plumbing objects (job_run, notification_*, inbox_message, http_delivery, ai_traces, automation_run, device_code, job) from the audit+activity writer. Kills the dominant amplifier. - driver-sql: default SQLite auto_vacuum=INCREMENTAL so freed pages can be reclaimed (Reaper hook lands in P1). - app-showcase: digest interval 20s→60s, flagged demo-only with cost rationale. Verified on a fresh showcase boot: 0 invalid_transition errors; over ~165s the digest still fires (job_run/notification/inbox +2 each) but sys_audit_log and sys_activity no longer grow at all (were +120 each in a comparable window); auto_vacuum reads back as INCREMENTAL. plugin-audit tests 18/18 pass; full build 69/69. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
documentation
Contributor
📓 Docs Drift CheckThis PR changes 2 package(s): 11 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
os-zhuang
added a commit
that referenced
this pull request
Jul 11, 2026
…Archiver, datasource separation, governance (#2786) (#2791) * docs(adr): ADR-0057 header/rollout sync — P0 shipped (#2079), P1–P4 tracked in #2786 The ADR header still said 'Proposed' with no pointer to the shipped P0 implementation or the new P1–P4 tracking issue. Per repo convention (cf. ADR-0030/0055/0069 headers), reflect implementation reality: - Status: Accepted — P0 shipped with this ADR (#2079); P1–P4 proposed, tracked in #2786 - §4 Rollout: link #2079 on the P0 row; add the #2786 tracking pointer Doc-only change; no code or policy content altered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNBzMWmSECrbiEDdVzwBt3 * feat(lifecycle): ADR-0057 P1 — lifecycle contract, LifecycleService Reaper, liveness gate, dogfood growth gate (#2786) The platform can declare an object's structure, validations and permissions — but not how long its data lives. P1 makes data lifecycle a first-class, declarable, runtime-enforced metadata concern: spec: - `lifecycle` object property (LifecycleSchema): class (record | audit | telemetry | transient | event) + retention.maxAge / ttl / storage(rotation) / archive / reclaim, with duration literals ('6h', '14d', '7y') - §3.5 enforce-or-remove invariants in a superRefine: a non-record class with no bounding policy is rejected; record-class policies are rejected; the archive window must start where the hot window ends - IDataDriver.reclaimSpace?() contract for post-sweep space reclamation objectql: - LifecycleService (Reaper), registered by ObjectQLPlugin as a platform service (default-on, OS_LIFECYCLE_DISABLED=1 / lifecycle.enabled=false to opt out): hourly sweep deletes rows past retention.maxAge (created_at) and past ttl.field + expireAfter, bounds rotation-declared telemetry by shards×unit until the P2 Rotator lands, and never hot-deletes an archive-declared audit ledger before the Archiver copies it out - Sweeps run bulk multi-deletes under a system context — at most ONE afterDelete hook per object per sweep, and telemetry sys_* objects are in the audit writer's SKIP_OBJECTS, so cleanup never re-feeds the tables it drains (the ADR-0052 self-audit trap) driver-sql: - reclaimSpace(): SQLite `PRAGMA incremental_vacuum` (the P0 ADR text claimed this hook shipped; it had not — this closes it), no-op on Postgres/MySQL platform objects (11 annotated): sys_activity 14d+rotation, sys_audit_log 90d→archive 7y, sys_metadata_audit 365d→archive, sys_job_run/sys_automation_run/sys_http_delivery 30d, notification pipeline (sys_notification / delivery / receipt / inbox) one 90d window, sys_device_code expires_at+1d. gates: - spec-liveness: object/lifecycle classified + bound to a dogfood proof (HIGH_RISK_CLASSES 'data-lifecycle') - dogfood storage-growth gate (@proof: adr0057-lifecycle-bounded-growth): boots showcase, proves contract coverage, reaps a backdated runaway writer, asserts record-class data untouched and archive-declared ledgers retained Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNBzMWmSECrbiEDdVzwBt3 * feat(lifecycle): ADR-0057 P2 — Rotator: physical time-sharding with O(1) shard DROP (#2786) High-frequency telemetry declared with lifecycle.storage.strategy='rotation' is now physically time-sharded on SQLite (the ServiceNow Table Rotation model, ADR-0057 §3.3): driver-sql: - rotateShards(objectDef, nowMs): idempotent Rotator entry — ensures the current shard (<table>__r<key>: day/week/month UTC keys), adopts a legacy pre-rotation table as its first shard (no data loss), column-syncs every retained shard so the UNION stays uniform, DROPs shards past the shards×unit time window (O(1) reclaim), and rebuilds the read view - The base name becomes a READ-ONLY UNION ALL view (SQLite views reject writes and RETURNING), so every write path redirects shard-wise instead: create/bulkCreate/upsert target the current shard; by-id update/delete probe shards newest-first; updateMany/deleteMany/bulkDelete fan out and sum — reads (find/count/aggregate) are untouched - Per-shard bookkeeping aliases (datetime/date/json/boolean coercion, tenant scope, timestamp stamping) so a shard builder behaves exactly like the view - initObjects routes rotation-declared objects to the Rotator (the plain create/alter DDL path would collide with the view) - supportsRotation gates on the dialect; declared indexes get per-shard names objectql LifecycleService: - Rotator invocation each sweep when the driver supports it; the fallback age-based reap covers every other dialect so the declared bound holds everywhere, only the reclamation mechanics differ - An explicit retention.maxAge still trims inside live shards after rotation (shard drops are unit-granular; this also immediately bounds a legacy table the Rotator just adopted whole) - Dropped shards trigger the datasource incremental-vacuum pass Transient TTL expiry (the other P2 line item) shipped with the P1 Reaper — TTL and age reaping are one mechanism (§3.3). Proof: sql-driver-rotation.test.ts drives the full shard lifecycle (create → rotate → window drop → adoption → cross-shard writes); the dogfood storage-growth gate now shards a probe stream through the real engine→driver path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNBzMWmSECrbiEDdVzwBt3 * feat(lifecycle): ADR-0057 P3 — datasource separation + Archiver cold store (#2786) Separation (§3.6): objects whose lifecycle.class is telemetry / event / audit now route to a dedicated 'telemetry' datasource whenever one is registered — engine getDriver() step 3, between datasourceMapping rules and the package-manifest defaultDatasource. Opt-in purely by the datasource's existence: no 'telemetry' driver ⇒ resolution is byte-for-byte what it was. Even on SQLite that is a separate file, so platform-generated growth can never again pollute the business DB. 'transient' deliberately stays on the primary: those objects are user-session data and some (better-auth's sys_device_code) are accessed outside the engine — splitting their storage would split their brain. Archiver (§3.3): audit-class objects with 'archive' declared now get the full retain → archive → delete flow instead of the P1 retain-only skip: - copies rows past archive.after to the archive datasource in bounded batches (500 × 20/sweep — a backlog drains across sweeps), as per-row idempotent upserts so an interrupted sweep re-converges - hot-deletes exactly the copied ids (bulkDelete), then the reclaim pass vacuums the hot store - archive.keep prunes the cold store itself - the cold schema is mirrored via idempotent syncSchema - no archive datasource registered ⇒ rows retained, object reported 'archive-pending' — a compliance ledger is never dropped unarchived Proof: engine-lifecycle-datasource.test.ts (routing incl. explicit-datasource precedence), Archiver unit tests (copy/delete id-exactness, keep-pruning, missing-datasource retention), and a dogfood case that provisions a real second SQL store mid-run and watches the ledger drain hot → cold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNBzMWmSECrbiEDdVzwBt3 * feat(lifecycle): ADR-0057 P4 — governance: quotas, growth alerts, tenant retention overrides (#2786) Adds the operations layer over the P1-P3 enforcement machinery, resolved from a new 'lifecycle' settings namespace (registered by ObjectQLPlugin at kernel:ready when a SettingsService is present; absent settings = declared policies apply unmodified): - enabled — runtime master switch (on top of OS_LIFECYCLE_DISABLED) - retention_overrides — per-object window overrides { object: { maxAge, expireAfter } }. TENANT-scoped: the sweep enumerates organizations, and a tenant whose override is genuinely tenant-scoped (not inherited) gets its own cutoff on its own rows, while the global pass covers everyone else INCLUDING NULL-org rows (a bare $nin would silently skip them). This is ADR-0057 §3.2 verbatim: regulated tenants set years, dev sets days — the same knob at different settings. An unparseable override keeps the declared window (never fails open into unbounded growth). - quotas / quota_defaults — per-object and per-class row ceilings - growth_alert_rows — per-sweep growth spike threshold Quota and growth breaches are OBSERVE-AND-ALERT only (report.alerts + onAlert sink, default logger warning) — governance never deletes beyond the declared policy; an operator decides. Every settings read is best-effort and snapshotted once per sweep; the tenant scan is capped (200) and skipped entirely on single-tenant kernels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNBzMWmSECrbiEDdVzwBt3 * docs(adr)+chore: ADR-0057 marked P0–P4 implemented; changeset for the lifecycle release (#2786) - ADR-0057 header + §4 rollout table now record P1–P4 as implemented in #2791; §3.3 gains the archive-safety hard rule and §3.6 the as-implemented 'telemetry' datasource routing note (transient exclusion rationale). - Changeset: minor bumps across spec / objectql / driver-sql / driver-sqlite-wasm / platform-objects / metadata-core / service-messaging / service-automation / plugin-audit, documenting the bounded-by-default behavior change and the override knobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNBzMWmSECrbiEDdVzwBt3 * chore(spec): regenerate api-surface snapshot for the ADR-0057 lifecycle exports The 5 additions (LifecycleSchema / LifecycleClassSchema / Lifecycle / LifecycleClass / LIFECYCLE_DURATION_REGEX) are the intentional P1 spec surface; the api-surface gate flagged them as uncommitted snapshot drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNBzMWmSECrbiEDdVzwBt3 --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A 20-second scheduled digest flow in
app-showcasegrewdev.dbto 260 MB+ over a multi-daypnpm dev— all platform-generated telemetry, zero business data. Forensics:sys_job_run+sys_notification(+delivery +receipt) +sys_inbox_message— and every one was mirrored into bothsys_audit_logandsys_activity(~21 rows/tick, audit+activity ≈ 76%).auto_vacuum=NONE, so the file never shrinks even after deletes.Root causes, by leverage: (1) no retention contract on platform objects; (2) audit+activity dual-write amplifying every internal mutation; (3)
auto_vacuum=NONE; (4) demo-tuned 20s interval.What
docs/adr/0057-system-data-lifecycle-and-retention.md— makes data lifecycle a first-class, declarable, runtime-enforced metadata concern (classify → declare → enforce via a platform-ownedLifecycleServiceReaper/Rotator/Archiver → reclaim → gate). Builds on ADR-0052 (bounded contexts) along the orthogonal time axis it never specified. Benchmarked against Salesforce (Big Objects / Field Audit Trail), ServiceNow (Table Rotation / Cleaner), Dataverse (Elastic TTL / data lake), OutSystems, Mendix. Rollout P0→P4.P0 stopgap (this PR):
plugin-audit— exclude operational/plumbing objects (sys_job,sys_job_run,sys_automation_run,sys_notification*,sys_inbox_message,sys_http_delivery,ai_traces,sys_device_code) from the audit+activity writerSKIP_OBJECTS. Kills the dominant amplifier.driver-sql— default SQLiteauto_vacuum=INCREMENTALso freed pages can be reclaimed (Reaper hook lands in P1). New DBs only; existing files adopt it after oneVACUUM.app-showcase— digest interval 20s→60s, flagged demo-only with cost rationale.Verification
Fresh showcase boot in this branch:
invalid_transitionerrors (the red noise that started this investigation).sys_job_run/sys_notification/sys_notification_delivery/sys_notification_receipt/sys_inbox_messageeach +2 (2 ticks @ 60s).sys_audit_logandsys_activitygrew by 0 (were +120 each in a comparable pre-fix window).PRAGMA auto_vacuumreads back as2(INCREMENTAL).plugin-audittests 18/18 pass; fullturbo build69/69.Combined effect: ~21 rows/tick @ 20s → 5 rows/tick @ 60s (≈93% lower row-growth rate), and space is now reclaimable.
🤖 Generated with Claude Code