feat(platform): ADR-0057 system data lifecycle + P0 telemetry-growth stopgap

docs/adr/0057-system-data-lifecycle-and-retention.md

@@ -0,0 +1,236 @@
+# ADR-0057: System data has a lifecycle — declarative retention, rotation, and reclamation for platform-generated objects
+
+**Status**: Proposed (2026-06-20)
+**Deciders**: ObjectStack Protocol Architects
+**Builds on**: [ADR-0052](./0052-audit-is-not-the-activity-feed.md) (decomposed audit / activity / collaboration into bounded contexts — this ADR adds the *orthogonal* axis those contexts never specified: **how long each one lives and how its space is reclaimed**), [ADR-0049](./0049-no-unenforced-security-properties.md) (enforce-or-remove — a declared `retention` that drives no sweeper is dead surface; this ADR wires it to a runtime consumer), [ADR-0030](./0030-notification-platform-convergence.md) (notification objects are messaging-owned with their own lifecycle), [ADR-0021](./0021-analytics-dataset-semantic-layer.md) (precedent for moving event-shaped data off the primary OLTP store)
+**Consumers**: `@objectstack/spec` (`lifecycle` object property + `LifecycleClass`), `@objectstack/objectql` (LifecycleService: Reaper / Rotator / Archiver), `@objectstack/plugin-audit` (audit/activity exclusion of telemetry objects — shipped P0), `@objectstack/driver-sql` (`auto_vacuum=INCREMENTAL` default + incremental-vacuum hook — shipped P0), `@objectstack/dogfood` (storage-growth regression gate), platform spec-liveness gate
+**Pilot**: `app-showcase` (its 20s `showcase_scheduled_digest` flow grew `dev.db` to 260 MB+ over a multi-day `pnpm dev` — the symptom this ADR generalizes from)
+
+---
+
+## TL;DR
+
+The platform can declare an object's structure, validations, permissions, and
+relationships — but **not how long its data should live**. Every
+platform-generated object therefore defaults to *immortal*. Any high-frequency
+write source then grows the database without bound, and because the SQLite
+driver ships `auto_vacuum=NONE`, the file never shrinks even after rows are
+deleted.
+
+A 20-second scheduled digest flow in `app-showcase` demonstrated this: each tick
+fanned out to a job-run row, a notification (+ delivery + receipt), and an inbox
+message — **and every one of those writes was mirrored into both `sys_audit_log`
+and `sys_activity`** (~21 append-only rows/tick, of which audit+activity were
+~76%). Over a multi-day dev session this reached **260 MB of pure
+platform-generated telemetry — zero business data**.
+
+This ADR makes **data lifecycle a first-class, declarable, runtime-enforced
+metadata concern**, the same posture the platform already takes for validation
+and security:
+
+1. **Classify** every object by `lifecycleClass`: `record` | `audit` |
+   `telemetry` | `transient` | `event`.
+2. **Declare** retention / rotation / archival on the object metadata.
+3. **Enforce** it with a single platform-owned **LifecycleService**
+   (Reaper + Rotator + Archiver) — not N per-plugin implementations.
+4. **Reclaim** space (SQLite `auto_vacuum=INCREMENTAL` + incremental vacuum).
+5. **Stop the amplifier**: operational telemetry is excluded from the
+   audit+activity writer at the seam (ADR-0052's event-spine finishes this).
+6. **Gate it**: spec-liveness requires a `lifecycle` declaration on every
+   non-`record` system object; a dogfood test asserts bounded growth so this
+   class of regression turns CI red instead of filling a disk.
+
+---
+
+## 1. Context — how a few seed records became 260 MB
+
+Forensics on the showcase `dev.db`:
+
+- The file was **261 MB**; a freshly seeded one is **2.3 MB**. ~110×.
+- **No business table grew.** `showcase_*`, `crm_*`, `pm_*` are the same seed
+  rows every boot (upsert overwrites). The growth was entirely in append-only
+  platform tables.
+- **Driver pragmas:** `auto_vacuum=NONE`, `journal_mode=delete`,
+  `freelist_count=0` → freed pages are never returned to the OS; the file is
+  pinned at its high-water mark.
+- **Two always-on scheduled jobs** drove the writes:
+  `showcase_scheduled_digest` (`interval`, **20 000 ms**) and
+  `approvals-sla-escalation` (`interval`, 300 000 ms).
+- **Measured fan-out** (controlled ~120 s run): `sys_audit_log` +120,
+  `sys_activity` +120, and `sys_job_run` / `sys_notification` /
+  `sys_notification_delivery` / `sys_notification_receipt` / `sys_inbox_message`
+  +15 each. Audit+activity were **~76 %** of all new rows — the dual-write
+  amplifier described in ADR-0052 §1.
+- **Rate:** ~+400 KB / 120 s ≈ **~290 MB/day** of continuous running.
+
+Root causes, in order of leverage:
+
+| # | Cause | Effect |
+|---|-------|--------|
+| 1 | No retention contract on any platform-generated object | append-only tables grow forever |
+| 2 | Audit+activity dual-write on *every* mutation incl. internal plumbing | 4–8× row amplification |
+| 3 | `auto_vacuum=NONE` | deleting rows would not shrink the file anyway |
+| 4 | Demo-tuned 20 s interval | turns a slow leak into a fast one |
+
+ADR-0052 already named #2 as a design flaw and set the long-term fix (an event
+spine; audit becomes a pure sink). This ADR addresses #1, #3 — the lifecycle
+axis — and ships the immediate mitigations for #2 and #4.
+
+## 2. The principle — telemetry is not a system of record
+
+The platform conflates two data kinds with opposite contracts into one OLTP
+store with one persistence policy:
+
+| Kind | Examples | True contract |
+|------|----------|---------------|
+| **System of record** | account, project, invoice, `sys_audit_log` | durable, retained, often immutable |
+| **Operational telemetry** | activity, job_run, notification_*, inbox, http_delivery, ai_traces | value decays with time; **must be bounded** |
+
+Every mature low-code platform separates these and bounds the second:
+
+| Platform | Mechanism |
+|---|---|
+| **Salesforce** | **Big Objects** for high-volume/historical; **Field Audit Trail** `HistoryRetentionPolicy` (archive ≤10y); **Platform Events** retained ≤72h |
+| **ServiceNow** | **Table Rotation** (time-sharded, DROP oldest shard = O(1) reclaim) + **Table Cleaner** (`sys_auto_flush`, delete by table/field/age) |
+| **Dataverse / Power Platform** | **Elastic tables** with native **TTL** column; **Bulk Deletion** jobs; configurable **audit retention**; long-term retention to a data lake |
+| **OutSystems** | logs in a **separate database** with configurable retention + cleanup |
+| **Mendix** | **non-persistable entities** (memory-only) for transient state |
+
+The consistent answer: **declarative, bounded-by-default lifecycle, enforced by a
+platform-owned sweeper, with space actually reclaimed.**
+
+## 3. Decision
+
+### 3.1 Classify — `lifecycleClass` on every object
+
+| Class | Meaning | Examples | Default contract |
+|---|---|---|---|
+| `record` | business truth | account, project, invoice | permanent, recoverable |
+| `audit` | compliance ledger | `sys_audit_log`, `sys_metadata_audit` | retain → archive → delete |
+| `telemetry` | high-frequency log / run flow | `sys_activity`, `sys_job_run`, `sys_notification_delivery`, `ai_traces`, `sys_http_delivery` | **rotation**, short retention |
+| `transient` | workflow / ephemeral state | `sys_notification_receipt`, read inbox, `sys_device_code` | **TTL** auto-expire |
+| `event` | event-bus messages | scheduled/trigger fan-out | very short TTL (hours) |
+
+`record` is the safe default (back-compat: undeclared objects keep today's
+behavior). The spec-liveness gate (§3.5) requires an explicit `lifecycle` on any
+object declared `audit`/`telemetry`/`transient`/`event`.
+
+### 3.2 Declare — `lifecycle` object metadata
+
+```ts
+defineObject({
+  name: 'sys_activity',
+  lifecycle: {
+    class: 'telemetry',
+    retention: { maxAge: '14d' },
+    storage:  { strategy: 'rotation', shards: 14, unit: 'day' }, // DROP oldest shard
+    reclaim: true,
+  },
+})
+
+defineObject({
+  name: 'sys_audit_log',
+  lifecycle: {
+    class: 'audit',
+    retention: { maxAge: '90d' },              // hot window
+    archive:   { after: '90d', to: 'datalake', keep: '7y' },
+    strategy: 'archive-then-delete',
+  },
+})
+
+defineObject({
+  name: 'sys_notification_receipt',
+  lifecycle: {
+    class: 'transient',
+    ttl: { field: 'created_at', expireAfter: '7d' },
+  },
+})
+```
+
+Policies are overridable per environment / tenant via `SettingsServicePlugin`
+(regulated tenants set years; dev sets days).
+
+### 3.3 Enforce — one platform-owned LifecycleService
+
+```
+LifecycleService (scans every object carrying a `lifecycle` declaration)
+├── Reaper    — delete by TTL/age in batches (transient + low-freq telemetry); then incremental_vacuum   [≈ ServiceNow Table Cleaner]
+├── Rotator   — time-shard high-freq telemetry; rotate by DROPping the oldest shard (O(1), real reclaim)  [≈ ServiceNow Table Rotation]
+└── Archiver  — copy audit cold data to an archive datasource, then delete from the hot store             [≈ SF Field Audit Trail / Dataverse data lake]
+```
+
+Hard rule: **the LifecycleService's own deletes/rotations are not audited or
+activity-logged** (otherwise cleanup re-feeds the tables it is draining — the
+same self-audit trap ADR-0052 already guards). Aggregate one summary row at
+most.
+
+### 3.4 Reclaim — driver space hygiene
+
+SQLite driver defaults to `auto_vacuum=INCREMENTAL` (shipped P0); the Reaper
+issues `PRAGMA incremental_vacuum` after a sweep. `auto_vacuum` only re-lays-out
+a *fresh* DB (or one after a one-time `VACUUM`), so existing files need a single
+`VACUUM` to adopt it — acceptable, since the Reaper / a `db:clean` covers legacy
+files.
+
+### 3.5 Gate — make the regression impossible to reintroduce
+
+- **Spec-liveness**: any object with `class ∈ {audit, telemetry, transient,
+  event}` and no `lifecycle.retention`/`ttl`/`rotation` ⇒ CI red. (Mirrors
+  ADR-0049 enforce-or-remove: a zero-retention telemetry object is a false
+  surface.)
+- **Dogfood storage-growth gate** (`@objectstack/dogfood`): boot an example
+  app, let scheduled timers tick N times, assert each telemetry table ≤ its
+  bound and DB-file delta ≤ threshold. Reverting any retention policy makes it
+  red — the golden test for this class.
+
+### 3.6 Physically separate telemetry (target state)
+
+`telemetry`/`audit`/`event` objects move to a dedicated `datasource` (the
+platform already supports `defaultDatasource`). Even on SQLite that is a
+separate file, so telemetry bloat can never again pollute the business DB, and
+the store can later be swapped for an append-only log / time-series / object
+store. This is the ADR-0021 dataset-migration pattern applied to event-shaped
+system data.
+
+## 4. Rollout
+
+| Phase | Scope | Status |
+|---|---|---|
+| **P0 — stop the bleed** | (a) exclude operational/plumbing objects from the audit+activity writer (`plugin-audit` `SKIP_OBJECTS`); (b) SQLite `auto_vacuum=INCREMENTAL` driver default; (c) showcase digest interval 20s→60s, flagged demo-only | **this ADR ships P0** |
+| **P1 — contract** | `lifecycleClass` + `lifecycle` spec; LifecycleService Reaper; spec-liveness enforcement; dogfood growth gate | proposed |
+| **P2 — rotation/TTL** | Rotator (shard + DROP) for high-freq telemetry; transient TTL expiry | proposed |
+| **P3 — separation** | telemetry/audit on a dedicated datasource; Archiver cold-store | proposed |
+| **P4 — governance** | per-table storage quotas, growth alerts, tenant-level retention overrides | proposed |
+
+P0 alone removes ~76 % of the row growth (audit/activity exclusion) and a
+further 3× (interval), and lets space be reclaimed — without any schema change
+or migration. P1+ makes it bounded by construction.
+
+## 5. Consequences
+
+- **Positive.** Platform-generated data is bounded by default and reclaimable.
+  Retention becomes a one-line declaration, enforced uniformly. Audit becomes a
+  clean ledger (finishing ADR-0052's direction). The 260 MB regression becomes a
+  CI assertion. Compliance retention (years) and dev ergonomics (days) are the
+  same knob at different settings.
+- **Cost.** A new spec property + a platform service + a datasource split.
+  Sequenced P0→P4 so each step is independently shippable and back-compatible.
+- **Back-compat.** Undeclared objects default to `record` (today's behavior).
+  Object names and schemas are unchanged; only lifecycle policy and (P3) owning
+  datasource change. The P0 audit exclusion only *stops creating* telemetry
+  audit/activity rows — it neither reads nor deletes existing ones.
+
+## 6. Alternatives considered
+
+- **Just delete `dev.db` periodically.** Treats the symptom; production and
+  long-lived dev DBs still grow unbounded, and nothing stops the next
+  high-frequency writer.
+- **Skip auditing by `context.isSystem`.** Too broad — the seed loader and
+  server-side automations write *business* records with `isSystem: true`;
+  dropping those from the ledger is a real compliance hole. The precise axis is
+  the object's lifecycle class, not who wrote it.
+- **Per-plugin cleanup jobs.** Produces N inconsistent implementations and
+  re-introduces the self-audit trap each time. Lifecycle is a platform
+  primitive, owned once.
+- **Lower the interval only.** Reduces the rate but leaves growth unbounded over
+  time and does not reclaim space.

examples/app-showcase/src/flows/index.ts

@@ -311,11 +311,16 @@ export const ScheduledDigestFlow = defineFlow({
     {
       id: 'start',
       type: 'start',
-      label: 'Every 20s',
+      label: 'Every 60s (demo)',
       config: {
-        // Interval keeps the demo observable in-session; production digests
-        // would use a cron expression, e.g. { type: 'cron', expression: '0 8 * * *' }.
-        schedule: { type: 'interval', intervalMs: 20000 },
+        // DEMO-ONLY interval. Each tick fans out into job_run + notification +
+        // delivery + receipt + inbox rows (all append-only, ADR-0057). At 20s
+        // this filled dev.db to 260MB+ over a multi-day `pnpm dev`. 60s keeps
+        // the schedule trigger observable within a minute while cutting the
+        // write rate 3x. Production digests use a cron expression instead,
+        // e.g. { type: 'cron', expression: '0 8 * * *' }. Real bounding comes
+        // from the lifecycle/retention work (ADR-0057), not this number.
+        schedule: { type: 'interval', intervalMs: 60000 },
       },
     },
     {

packages/plugins/driver-sql/src/sql-driver.ts

@@ -391,6 +391,23 @@ export class SqlDriver implements IDataDriver {
     // Ensure the database directory exists before any query can trigger
     // better-sqlite3 to open the file (e.g. loadMetaFromDb on startup).
     await this.ensureDatabaseExists();
+
+    // SQLite space hygiene (ADR-0057). With the default `auto_vacuum=NONE`,
+    // freed pages are never returned to the OS — a database that briefly grew
+    // (e.g. high-frequency telemetry before retention sweeps run) stays at its
+    // high-water mark forever. INCREMENTAL lets a later `PRAGMA incremental_vacuum`
+    // (run by the lifecycle Reaper, or manually) reclaim that space without a
+    // full blocking VACUUM. NOTE: auto_vacuum only changes layout on a *fresh*
+    // database or after a one-time full VACUUM, so this benefits new dev DBs;
+    // existing files need a single `VACUUM` to adopt it. Harmless / no-op on
+    // :memory: and on already-incremental databases.
+    if (this.isSqlite) {
+      try {
+        await this.knex.raw('PRAGMA auto_vacuum = INCREMENTAL');
+      } catch (e) {
+        this.logger.warn('Failed to set PRAGMA auto_vacuum=INCREMENTAL', e);
+      }
+    }
   }
 
   async checkHealth(): Promise<boolean> {

packages/plugins/plugin-audit/src/audit-writers.ts

@@ -54,8 +54,23 @@ export interface AuditWriterOptions {
  * permissions and always succeed regardless of the calling user's RBAC.
  */
 
-/** Tables that are intentionally excluded from audit/activity writes. */
+/**
+ * Tables that are intentionally excluded from audit/activity writes.
+ *
+ * Two reasons an object lands here:
+ *  1. Recursion / auth noise — the audit & activity tables themselves, plus
+ *     session/presence/auth tables (high-frequency, low value).
+ *  2. Operational telemetry / plumbing (ADR-0057) — platform-internal
+ *     event/log/queue objects with a `telemetry`/`transient`/`event`
+ *     lifecycle class. These are NOT user-attributable, compliance-relevant
+ *     changes; mirroring every one of them into the immutable audit ledger
+ *     *and* the activity feed is the dominant source of unbounded growth (a
+ *     single 20s scheduled flow fanned out to ~21 audit+activity rows/tick,
+ *     ~76% of all row growth — see the lifecycle-retention ADR). Until the
+ *     event-spine (ADR-0052 §P2) lands, exclude them at the writer seam.
+ */
 const SKIP_OBJECTS = new Set<string>([
+  // (1) recursion + auth/session noise
   'sys_audit_log',
   'sys_activity',
   'sys_comment',
@@ -65,6 +80,17 @@ const SKIP_OBJECTS = new Set<string>([
   'sys_account_session',
   'sys_account_verification',
   'sys_account_account',
+  'sys_device_code',
+  // (2) operational telemetry / plumbing (ADR-0057 — telemetry/transient/event)
+  'sys_job',                     // schedule heartbeats (last_run_at churn)
+  'sys_job_run',                 // one row per scheduled execution
+  'sys_automation_run',          // one row per automation execution
+  'sys_notification',            // messaging-owned (ADR-0030); its own lifecycle
+  'sys_notification_delivery',
+  'sys_notification_receipt',
+  'sys_inbox_message',           // per-user fan-out of every notification
+  'sys_http_delivery',           // webhook/outbound transport log
+  'ai_traces',                   // LLM trace telemetry
 ]);
 
 /** Fields that are noise in diffs (always change, never user-meaningful). */