Skip to content

Commit 81ce41a

Browse files
authored
feat(rest): preserve the original audit timeline for a historical import (#3493) (#3497)
Follow-up to #3479/#3483. Adds opt-in ExecutionContext.preserveAudit (set by the import runner on treatAsHistorical) so a historical import keeps the original timeline: the audit hook treats updated_at/updated_by as client-preferred, the readonly strip admits a whitelist (audit family + author-declared business readonly fields) while still stripping platform-managed system columns, and the SQL driver keeps a supplied updated_at instead of force-stamping now. Fully opt-in; normal writes unchanged.
1 parent 23624df commit 81ce41a

14 files changed

Lines changed: 352 additions & 18 deletions

File tree

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
"@objectstack/spec": patch
3+
"@objectstack/objectql": patch
4+
"@objectstack/driver-sql": patch
5+
"@objectstack/rest": patch
6+
---
7+
8+
feat(rest): `treatAsHistorical` import also preserves the original audit timeline (#3493)
9+
10+
Follow-up to #3479/#3483. `treatAsHistorical` solved the FSM half — mid-lifecycle
11+
rows are no longer rejected by `initialStates` — but the OTHER half of a historical
12+
migration, preserving the original timeline, still didn't hold: an imported ticket
13+
that closed in 2021 stored `updated_at` = the import day (and `updated_by` = the
14+
importer), and a `writeMode: 'upsert'` refresh silently dropped business `readonly`
15+
fields (`closed_at`, `resolved_by`). Reports, audit, and "recently modified"
16+
sorting all came out wrong.
17+
18+
Three layers were force-overwriting the timeline; all three now respect a single
19+
new opt-in flag, `ExecutionContext.preserveAudit`, which `treatAsHistorical` sets
20+
alongside `skipStateMachine`:
21+
22+
- **spec**: `ExecutionContext.preserveAudit` (server-set only, never client-supplied)
23+
and `DriverOptions.preserveAudit` (threaded to the driver's update stamp).
24+
- **objectql** — the built-in audit hook (`plugin.ts`) now treats `updated_at` /
25+
`updated_by` as CLIENT-PREFERRED (`?? now` / `?? userId`) under `preserveAudit`,
26+
symmetric with how `created_at` / `created_by` already behave on insert; and the
27+
static-`readonly` write strip (`stripReadonlyFields`) admits a WHITELIST — the
28+
audit/timestamp family plus author-declared business `readonly` fields — so an
29+
upsert refresh no longer drops them.
30+
- **driver-sql** — the SQL `update` path keeps a supplied `updated_at` instead of
31+
force-advancing it to `now` when `DriverOptions.preserveAudit` is set (fills-only-
32+
empty, mirroring the insert stamp).
33+
- **rest** — the import runner sets `preserveAudit` on the write context iff the
34+
request opts into `treatAsHistorical`.
35+
36+
Deliberately a WHITELIST, not the blanket `isSystem` exemption: platform-managed
37+
`system` columns OUTSIDE the audit family (`organization_id` / tenancy, generated
38+
columns) STAY stripped, so a historical import reinstates established facts without
39+
becoming a backdoor to forge tenancy. Permissions / RLS / field-level security are
40+
unaffected — this changes only which audit/readonly values the runtime overwrites,
41+
never who may write the record. Fully opt-in: a normal write still auto-stamps
42+
`updated_at`/`updated_by` and strips `readonly` exactly as before. The objectui
43+
"Import as historical data" checkbox (objectui#2815) now drives both halves — no new
44+
UI.

content/docs/references/data/driver.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ const result = DriverCapabilities.parse(data);
9696
| **traceContext** | `Record<string, string>` | optional | OpenTelemetry context or request ID |
9797
| **tenantId** | `string` | optional | Tenant Isolation identifier |
9898
| **timezone** | `string` | optional | Business reference timezone (IANA) for date-dependent generation, e.g. autonumber date tokens |
99+
| **preserveAudit** | `boolean` | optional | Historical import: keep a supplied updated_at instead of force-stamping now (from ExecutionContext.preserveAudit) |
99100

100101

101102
---

content/docs/references/kernel/execution-context.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ const result = ExecutionContext.parse(data);
6969
| **skipAutomations** | `boolean` | optional | |
7070
| **seedReplay** | `boolean` | optional | |
7171
| **skipStateMachine** | `boolean` | optional | |
72+
| **preserveAudit** | `boolean` | optional | |
7273
| **oauthScopes** | `string[]` | optional | |
7374
| **accessToken** | `string` | optional | |
7475
| **transaction** | `any` | optional | |

packages/objectql/src/engine.ts

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -782,6 +782,10 @@ export class ObjectQL implements IDataEngine {
782782
// Propagate the full automation opt-out so `triggerHooks` can skip
783783
// metadata-bound hooks (import with "run automations" unchecked, undo).
784784
...((execCtx as any).skipAutomations ? { skipAutomations: true } : {}),
785+
// Propagate the historical-import audit-preservation flag so the built-in
786+
// audit hook keeps a client-supplied updated_at/updated_by instead of
787+
// stamping now (#3493). Opt-in, server-set only.
788+
...((execCtx as any).preserveAudit ? { preserveAudit: true } : {}),
785789
} as HookContext['session'];
786790
}
787791

@@ -858,7 +862,8 @@ export class ObjectQL implements IDataEngine {
858862
!isTenancyDisabled(this._registry.getObject(object));
859863
const hasTz = execCtx?.timezone !== undefined;
860864
const isSystem = execCtx?.isSystem === true;
861-
if (!hasTx && !hasTenant && !isSystem && !hasTz) return base;
865+
const preserveAudit = (execCtx as any)?.preserveAudit === true;
866+
if (!hasTx && !hasTenant && !isSystem && !hasTz && !preserveAudit) return base;
862867
const opts: any = base && typeof base === 'object' ? { ...base } : {};
863868
if (hasTx && opts.transaction === undefined) {
864869
opts.transaction = tx;
@@ -877,6 +882,11 @@ export class ObjectQL implements IDataEngine {
877882
// still flag genuine user-path bugs.
878883
opts.bypassTenantAudit = true;
879884
}
885+
if (preserveAudit && opts.preserveAudit === undefined) {
886+
// Historical import (#3493): let the driver keep a supplied `updated_at`
887+
// instead of force-stamping now on the update path.
888+
opts.preserveAudit = true;
889+
}
880890
return opts;
881891
}
882892

@@ -2868,7 +2878,7 @@ export class ObjectQL implements IDataEngine {
28682878
// read-only writes are dropped, never the server stamps.
28692879
if (!opCtx.context?.isSystem) {
28702880
const preRo = hookContext.input.data as Record<string, unknown>;
2871-
hookContext.input.data = stripReadonlyFields(updateSchema as any, preRo, suppliedKeys, this.logger) as any;
2881+
hookContext.input.data = stripReadonlyFields(updateSchema as any, preRo, suppliedKeys, this.logger, { preserveAudit: opCtx.context?.preserveAudit === true }) as any;
28722882
reportDroppedFields(preRo, hookContext.input.data as Record<string, unknown>, 'readonly');
28732883
}
28742884
evaluateValidationRules(updateSchema as any, hookContext.input.data as Record<string, unknown>, 'update', { previous: priorRecord, logger: this.logger, currentUser: this.buildEvalUser(opCtx.context), skipStateMachine: shouldSkipStateMachine(opCtx.context) });
@@ -2923,7 +2933,7 @@ export class ObjectQL implements IDataEngine {
29232933
// rejected upstream by the tenant write wall, #2946).
29242934
if (!opCtx.context?.isSystem) {
29252935
const preRoMulti = hookContext.input.data as Record<string, unknown>;
2926-
hookContext.input.data = stripReadonlyFields(updateSchema as any, preRoMulti, suppliedKeys, this.logger) as any;
2936+
hookContext.input.data = stripReadonlyFields(updateSchema as any, preRoMulti, suppliedKeys, this.logger, { preserveAudit: opCtx.context?.preserveAudit === true }) as any;
29272937
reportDroppedFields(preRoMulti, hookContext.input.data as Record<string, unknown>, 'readonly');
29282938
}
29292939
// [#3106] Same enforcement the single-id branch runs at its

packages/objectql/src/plugin.integration.test.ts

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1380,6 +1380,75 @@ describe('ObjectQLPlugin - Metadata Service Integration', () => {
13801380
});
13811381
});
13821382

1383+
// #3493 — an opt-in "historical" import (context.preserveAudit) reinstates the
1384+
// ORIGINAL timeline on UPDATE: the audit hook keeps a client-supplied
1385+
// updated_at/updated_by (instead of stamping now/importer), and the readonly
1386+
// strip admits the audit family + author-declared business readonly fields
1387+
// (closed_at) instead of dropping them. A normal write (no flag) still stamps
1388+
// and strips. (The driver's own updated_at force-stamp is covered separately in
1389+
// driver-sql's timestamp-format test — this mock driver echoes the data.)
1390+
describe('preserveAudit — historical import keeps the original timeline on UPDATE (#3493)', () => {
1391+
async function bootHistorical() {
1392+
const updates: Record<string, any>[] = [];
1393+
const mockDriver = {
1394+
name: 'hist-capture', version: '1.0.0',
1395+
connect: async () => {}, disconnect: async () => {},
1396+
find: async () => [], findOne: async () => null,
1397+
create: async (_o: string, d: any) => ({ id: 'rec-1', ...d }),
1398+
update: async (_o: string, _i: any, d: any) => { updates.push({ ...d }); return { id: _i, ...d }; },
1399+
delete: async () => true, syncSchema: async () => {},
1400+
};
1401+
await kernel.use({
1402+
name: 'hist-capture-plugin', type: 'driver', version: '1.0.0',
1403+
init: async (ctx) => { ctx.registerService('driver.hist-capture', mockDriver); },
1404+
});
1405+
await kernel.use(new ObjectQLPlugin());
1406+
await kernel.bootstrap();
1407+
const objectql = kernel.getService('objectql') as any;
1408+
const obj: ObjectSchema = {
1409+
name: 'ticket_obj', label: 'Ticket', datasource: 'hist-capture',
1410+
fields: {
1411+
name: { name: 'name', label: 'Name', type: 'text' },
1412+
updated_by: { name: 'updated_by', label: 'Updated By', type: 'text', readonly: true } as any,
1413+
// author-declared business readonly field — a case's close time
1414+
closed_at: { name: 'closed_at', label: 'Closed At', type: 'datetime', readonly: true } as any,
1415+
},
1416+
};
1417+
objectql.registry.registerObject(obj, 'test', 'test');
1418+
return { objectql, updates };
1419+
}
1420+
1421+
it('KEEPS a supplied updated_at / updated_by / closed_at under preserveAudit', async () => {
1422+
const { objectql, updates } = await bootHistorical();
1423+
const ts = '2021-03-01T09:00:00.000Z';
1424+
await objectql.update(
1425+
'ticket_obj',
1426+
{ id: 'rec-1', name: 'B', updated_at: ts, updated_by: 'u_original', closed_at: ts },
1427+
{ context: { userId: 'u_import', preserveAudit: true } },
1428+
);
1429+
expect(updates.length).toBe(1);
1430+
const data = updates[0];
1431+
expect(data.name).toBe('B');
1432+
expect(data.updated_at).toBe(ts); // hook kept client value (not "now")
1433+
expect(data.updated_by).toBe('u_original'); // hook kept client value (not importer)
1434+
expect(data.closed_at).toBe(ts); // business readonly reinstated, not stripped
1435+
});
1436+
1437+
it('a normal update (no preserveAudit) still overwrites updated_by and strips closed_at', async () => {
1438+
const { objectql, updates } = await bootHistorical();
1439+
await objectql.update(
1440+
'ticket_obj',
1441+
{ id: 'rec-1', name: 'B', closed_at: '2021-03-01T09:00:00.000Z' },
1442+
{ context: { userId: 'u_import' } },
1443+
);
1444+
expect(updates.length).toBe(1);
1445+
const data = updates[0];
1446+
expect(data.name).toBe('B');
1447+
expect(data.updated_by).toBe('u_import'); // server-stamped to the actor
1448+
expect(data).not.toHaveProperty('closed_at'); // readonly business field stripped
1449+
});
1450+
});
1451+
13831452
// #3042 — conditional `readonlyWhen` must be enforced on the BULK
13841453
// (updateMany) path too, not only the single-id path. The bulk strip reads
13851454
// the matched rows' prior state and drops a field locked in ≥1 of them.

packages/objectql/src/plugin.ts

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -759,16 +759,22 @@ export class ObjectQLPlugin implements Plugin {
759759
isInsert: boolean,
760760
) => {
761761
const now = stamp();
762+
// A "historical" import (#3493) reinstates the ORIGINAL timeline, so a
763+
// client-supplied updated_at/updated_by is CLIENT-PREFERRED here —
764+
// symmetric with created_at/created_by on insert — instead of being
765+
// overwritten with the import instant. Opt-in and server-set only; a
766+
// normal write leaves `preserveAudit` unset and still stamps now.
767+
const preserveAudit = session?.preserveAudit === true;
762768
if (isInsert) {
763769
record.created_at = record.created_at ?? now;
764770
}
765-
record.updated_at = now;
771+
record.updated_at = preserveAudit ? (record.updated_at ?? now) : now;
766772
if (session?.userId) {
767773
if (isInsert && hasField(objectName, 'created_by')) {
768774
record.created_by = record.created_by ?? session.userId;
769775
}
770776
if (hasField(objectName, 'updated_by')) {
771-
record.updated_by = session.userId;
777+
record.updated_by = preserveAudit ? (record.updated_by ?? session.userId) : session.userId;
772778
}
773779
}
774780
// Stamp the driver-layer `tenant_id` column from the caller's active org.

packages/objectql/src/validation/rule-validator.test.ts

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,72 @@ describe('stripReadonlyFields (#2948)', () => {
153153
});
154154
});
155155

156+
// #3493 — a "historical" import (preserveAudit) reinstates the original
157+
// timeline: the audit/timestamp family and author-declared business `readonly`
158+
// fields survive the strip, while platform-managed `system` columns outside the
159+
// audit family stay stripped (no tenancy-forging backdoor).
160+
const historicalFields = {
161+
fields: {
162+
title: { type: 'text' },
163+
created_at: { type: 'datetime', readonly: true, system: true },
164+
created_by: { type: 'lookup', readonly: true, system: true },
165+
updated_at: { type: 'datetime', readonly: true, system: true },
166+
updated_by: { type: 'lookup', readonly: true, system: true },
167+
organization_id: { type: 'lookup', readonly: true, system: true },
168+
// author-declared business readonly field (NOT system) — e.g. a case's close time
169+
closed_at: { type: 'datetime', readonly: true },
170+
},
171+
};
172+
173+
describe('stripReadonlyFields — preserveAudit whitelist (#3493)', () => {
174+
it('KEEPS the caller-supplied audit/timestamp family under preserveAudit', () => {
175+
const supplied = new Set(['created_at', 'created_by', 'updated_at', 'updated_by']);
176+
const data = {
177+
created_at: '2020-01-01T00:00:00Z',
178+
created_by: 'u_creator',
179+
updated_at: '2021-03-01T00:00:00Z',
180+
updated_by: 'u_old',
181+
};
182+
const out = stripReadonlyFields(historicalFields, { ...data }, supplied, undefined, { preserveAudit: true });
183+
expect(out).toEqual(data);
184+
});
185+
186+
it('KEEPS an author-declared business readonly field (closed_at) under preserveAudit', () => {
187+
const supplied = new Set(['closed_at']);
188+
const out = stripReadonlyFields(
189+
historicalFields,
190+
{ closed_at: '2021-03-01T00:00:00Z' },
191+
supplied,
192+
undefined,
193+
{ preserveAudit: true },
194+
);
195+
expect(out).toEqual({ closed_at: '2021-03-01T00:00:00Z' });
196+
});
197+
198+
it('STILL strips a non-audit system column (organization_id) under preserveAudit — no tenancy backdoor', () => {
199+
const supplied = new Set(['organization_id', 'closed_at']);
200+
const out = stripReadonlyFields(
201+
historicalFields,
202+
{ organization_id: 'org_forged', closed_at: '2021-03-01T00:00:00Z' },
203+
supplied,
204+
undefined,
205+
{ preserveAudit: true },
206+
);
207+
expect(out).toEqual({ closed_at: '2021-03-01T00:00:00Z' });
208+
});
209+
210+
it('strips the whole family as before when preserveAudit is NOT set (regression)', () => {
211+
const supplied = new Set(['updated_at', 'closed_at', 'organization_id']);
212+
const out = stripReadonlyFields(historicalFields, {
213+
title: 'x',
214+
updated_at: '2021-03-01T00:00:00Z',
215+
closed_at: '2021-03-01T00:00:00Z',
216+
organization_id: 'o1',
217+
}, supplied);
218+
expect(out).toEqual({ title: 'x' });
219+
});
220+
});
221+
156222
describe('needsPriorRecord — field conditional rules (B2)', () => {
157223
it('is true when a field declares requiredWhen / readonlyWhen', () => {
158224
expect(needsPriorRecord(invoiceFields as any)).toBe(true);

packages/objectql/src/validation/rule-validator.ts

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -325,6 +325,15 @@ export function stripReadonlyWhenFieldsMulti(
325325
* system-context writes (import, seed replay, approvals, lifecycle hooks —
326326
* all `isSystem: true`) legitimately set read-only columns and skip it.
327327
*
328+
* `options.preserveAudit` (#3493) relaxes the strip for an opt-in "historical"
329+
* import that reinstates the original timeline: a caller-supplied read-only
330+
* field is KEPT when {@link isPreservableUnderAudit} allows it — the
331+
* audit/timestamp family or any author-declared business `readonly` field.
332+
* Platform-managed `system` columns outside that family (`organization_id` /
333+
* tenancy, generated columns) stay stripped, so the relaxation reinstates facts
334+
* without becoming a tenancy-forging backdoor. It is a WHITELIST, deliberately
335+
* narrower than the blanket `isSystem` exemption above.
336+
*
328337
* Returns the same object when nothing is stripped, else a shallow copy with the
329338
* offending keys removed.
330339
*/
@@ -333,21 +342,54 @@ export function stripReadonlyFields(
333342
data: Record<string, unknown> | undefined | null,
334343
suppliedKeys: ReadonlySet<string>,
335344
logger?: EvaluateRulesOptions['logger'],
345+
options?: { preserveAudit?: boolean },
336346
): Record<string, unknown> | undefined | null {
337347
const fields = objectSchema?.fields;
338348
if (!fields || !data) return data;
349+
const preserveAudit = options?.preserveAudit === true;
339350
let result = data;
340351
for (const [name, def] of Object.entries(fields)) {
341352
if (!def?.readonly) continue;
342353
if (!(name in (result as Record<string, unknown>))) continue;
343354
if (!suppliedKeys.has(name)) continue; // server-stamped, not caller-supplied — keep
355+
if (preserveAudit && isPreservableUnderAudit(name, def)) continue; // historical import reinstates it
344356
if (result === data) result = { ...data };
345357
delete (result as Record<string, unknown>)[name];
346358
logger?.warn?.(`Field '${name}' is read-only — ignoring incoming change (#2948)`);
347359
}
348360
return result;
349361
}
350362

363+
/**
364+
* The audit / attribution family — the "original timeline" a historical import
365+
* (`preserveAudit`) is allowed to reinstate even though these columns are
366+
* `system` + `readonly`. Kept in sync with the registry's auto-injected audit
367+
* fields (`packages/objectql/src/registry.ts`).
368+
*/
369+
const AUDIT_TIMELINE_FIELDS: ReadonlySet<string> = new Set([
370+
'created_at',
371+
'created_by',
372+
'updated_at',
373+
'updated_by',
374+
]);
375+
376+
/**
377+
* Whether a caller-supplied `readonly` field may be REINSTATED by an opt-in
378+
* historical import (`preserveAudit`), rather than stripped (#3493). Allows:
379+
* - the audit/timestamp family (`created_*` / `updated_*`) — the timeline the
380+
* feature exists to preserve; and
381+
* - author-declared business `readonly` fields (`closed_at`, `resolved_by`, …),
382+
* i.e. anything NOT flagged `system: true`.
383+
* Still strips platform-managed `system` columns outside the audit family
384+
* (`organization_id` / tenancy, generated columns): a historical import may
385+
* reinstate established facts, but must not forge tenancy or system-generated
386+
* values.
387+
*/
388+
function isPreservableUnderAudit(name: string, def: ConditionalFieldDef): boolean {
389+
if (AUDIT_TIMELINE_FIELDS.has(name)) return true;
390+
return def.system !== true;
391+
}
392+
351393
/**
352394
* A rule needs the prior record if it reasons about the transition or compares
353395
* against unchanged fields (`state_machine` / `cross_field` / `script`), or if
@@ -383,6 +425,10 @@ interface ConditionalFieldDef {
383425
readonlyWhen?: string | Expression;
384426
/** Static, unconditional read-only flag (`field.readonly`). #2948. */
385427
readonly?: boolean;
428+
/** Auto-injected platform/audit field (`field.system`). Distinguishes the
429+
* audit family / tenancy columns from author-declared business fields when
430+
* a historical import (`preserveAudit`) relaxes the readonly strip. #3493. */
431+
system?: boolean;
386432
/** Field type — scopes per-option `visibleWhen` enforcement to choice fields. */
387433
type?: string;
388434
/** select/multiselect/radio/checkboxes options; an option may gate itself with `visibleWhen`. */

0 commit comments

Comments
 (0)