|
| 1 | +# Data Lifecycle & Retention |
| 2 | + |
| 3 | +Guide for declaring how long an object's data lives and how its space is |
| 4 | +reclaimed (ADR-0057). Not to be confused with **lifecycle hooks** |
| 5 | +(`beforeInsert` / `afterUpdate` …) — those run business logic on data |
| 6 | +operations; *this* page is about retention, rotation, and archival of the |
| 7 | +rows themselves. |
| 8 | + |
| 9 | +## Why This Exists |
| 10 | + |
| 11 | +Every object without a `lifecycle` block is **immortal**: rows are never |
| 12 | +deleted, and on SQLite the file never shrinks. That's correct for business |
| 13 | +records — and a guaranteed disk leak for anything append-only. A scheduled |
| 14 | +flow writing telemetry every 20 seconds once grew a dev database from 2 MB |
| 15 | +to 260 MB with zero business data in it. |
| 16 | + |
| 17 | +**Rule: any append-only, high-write-rate object (event log, run history, |
| 18 | +delivery outbox, ephemeral tokens) MUST declare a `lifecycle` block.** |
| 19 | +The platform LifecycleService sweeps declared policies hourly, deletes |
| 20 | +expired rows under a system context, and reclaims driver space. |
| 21 | + |
| 22 | +## Lifecycle Classes |
| 23 | + |
| 24 | +| Class | Contract | Typical use | |
| 25 | +|:------|:---------|:------------| |
| 26 | +| `record` | Business truth — permanent; policies FORBIDDEN | accounts, orders, invoices | |
| 27 | +| `audit` | Compliance ledger — retain → archive → delete | audit trails | |
| 28 | +| `telemetry` | High-frequency log — rotation / short retention | activity streams, run history | |
| 29 | +| `transient` | Ephemeral state — TTL auto-expire | receipts, codes, sessions | |
| 30 | +| `event` | Bus messages — very short TTL (hours) | scheduled fan-out | |
| 31 | + |
| 32 | +## Syntax |
| 33 | + |
| 34 | +```typescript |
| 35 | +import { ObjectSchema, Field } from '@objectstack/spec/data'; |
| 36 | + |
| 37 | +// ✅ Telemetry stream: bounded by a rotation window |
| 38 | +export const ApiCallLog = ObjectSchema.create({ |
| 39 | + name: 'api_call_log', |
| 40 | + lifecycle: { |
| 41 | + class: 'telemetry', |
| 42 | + retention: { maxAge: '14d' }, // reap past 14 days (created_at) |
| 43 | + storage: { strategy: 'rotation', shards: 14, unit: 'day' }, // O(1) shard DROP on SQLite |
| 44 | + }, |
| 45 | + fields: { |
| 46 | + endpoint: Field.text({}), |
| 47 | + status: Field.number({}), |
| 48 | + }, |
| 49 | +}); |
| 50 | + |
| 51 | +// ✅ Ephemeral token: TTL on its own expiry field |
| 52 | +export const ImportTicket = ObjectSchema.create({ |
| 53 | + name: 'import_ticket', |
| 54 | + lifecycle: { |
| 55 | + class: 'transient', |
| 56 | + ttl: { field: 'expires_at', expireAfter: '1d' }, // reap 1 day after expires_at |
| 57 | + }, |
| 58 | + fields: { |
| 59 | + expires_at: Field.datetime({}), |
| 60 | + }, |
| 61 | +}); |
| 62 | + |
| 63 | +// ✅ Compliance ledger: hot window, then cold storage |
| 64 | +export const ConsentLog = ObjectSchema.create({ |
| 65 | + name: 'consent_log', |
| 66 | + lifecycle: { |
| 67 | + class: 'audit', |
| 68 | + retention: { maxAge: '90d' }, |
| 69 | + archive: { after: '90d', to: 'archive', keep: '7y' }, // must: after === maxAge |
| 70 | + }, |
| 71 | + fields: { |
| 72 | + subject: Field.text({}), |
| 73 | + }, |
| 74 | +}); |
| 75 | +``` |
| 76 | + |
| 77 | +Duration literals: `<n>` + `h` / `d` / `w` / `y` — e.g. `'6h'`, `'14d'`, |
| 78 | +`'12w'`, `'7y'`. |
| 79 | + |
| 80 | +## Validation Rules (rejected at parse time) |
| 81 | + |
| 82 | +```typescript |
| 83 | +// ❌ Non-record class with no bounding policy — grows forever, exactly the bug |
| 84 | +lifecycle: { class: 'telemetry' } |
| 85 | + |
| 86 | +// ❌ Policies on record-class — business truth is permanent |
| 87 | +lifecycle: { class: 'record', retention: { maxAge: '30d' } } |
| 88 | + |
| 89 | +// ❌ Archive window detached from the hot window |
| 90 | +lifecycle: { class: 'audit', retention: { maxAge: '90d' }, archive: { after: '30d', to: 'archive' } } |
| 91 | + |
| 92 | +// ❌ Free-form durations |
| 93 | +lifecycle: { class: 'telemetry', retention: { maxAge: '2 weeks' } } // use '2w' |
| 94 | +``` |
| 95 | + |
| 96 | +## Safety Semantics |
| 97 | + |
| 98 | +- **No `lifecycle` block = today's behavior.** Nothing is deleted. `record` |
| 99 | + is the implicit default. |
| 100 | +- **Archive-then-delete is atomic-ish and safe by default**: an object |
| 101 | + declaring `archive` is never hot-deleted before the copy to the archive |
| 102 | + datasource succeeded. If no datasource is registered under the `archive.to` |
| 103 | + name, rows are simply retained — a compliance ledger cannot be destroyed |
| 104 | + by declaring a lifecycle. |
| 105 | +- **Rotation** physically time-shards the table on SQLite (writes hit the |
| 106 | + current shard, reads go through a view under the object's name, expired |
| 107 | + shards are DROPped). Other dialects enforce the same window with an |
| 108 | + age-based reap. |
| 109 | +- **Separation**: registering a datasource named `telemetry` moves every |
| 110 | + `telemetry`/`event`/`audit` object onto it — telemetry bloat can't touch |
| 111 | + the business DB. |
| 112 | + |
| 113 | +## Operations |
| 114 | + |
| 115 | +| Knob | Where | Effect | |
| 116 | +|:-----|:------|:-------| |
| 117 | +| `enabled` | `lifecycle` settings namespace | Runtime master switch | |
| 118 | +| `retention_overrides` | settings, **tenant-scoped** | Per-object window overrides — a regulated tenant sets `'2y'` while dev keeps days | |
| 119 | +| `quotas` / `quota_defaults` | settings | Row ceilings; breaches ALERT (never delete beyond declared policy) | |
| 120 | +| `growth_alert_rows` | settings | Per-sweep growth spike alert | |
| 121 | +| `OS_LIFECYCLE_DISABLED=1` | env | Disables sweeping entirely | |
| 122 | + |
| 123 | +## Decision Tree |
| 124 | + |
| 125 | +1. Users create/edit it and it represents business state? → `record` (omit the block). |
| 126 | +2. Is it a compliance/audit trail? → `audit` + `retention`, add `archive` if cold storage exists. |
| 127 | +3. Written by machines at high frequency, value decays in days/weeks? → `telemetry` + `retention` (add rotation `storage` for the hottest tables). |
| 128 | +4. Meaningless after a deadline (tokens, receipts, read-state)? → `transient` + `ttl` on the natural expiry field. |
| 129 | +5. Bus/fan-out messages? → `event` + a short `ttl` (hours). |
0 commit comments