Skip to content

Commit c80aece

Browse files
os-zhuangclaude
andauthored
test+docs: harden and document the historical-import preserveAudit seam (#3493 follow-ups) (#3560)
Follow-ups to #3493 / #3549 / #3556 — no behavior change. Real-SQLite end-to-end test (@objectstack/runtime): wires the real ObjectQL engine to the real SqlDriver (better-sqlite3) and proves the preserveAudit seam that mock / in-memory drivers structurally cannot — that context.preserveAudit threads through buildDriverOptions and defeats the SQL driver's updated_at force-stamp on both the forward historical write and the undo restore, and that a business readonly field (closed_at) survives the engine strip all the way to disk. Includes the #3549 undo capstone: restore-under-preserveAudit rolls the timeline back, while a plain restore re-stamps now (the bug #3556 fixed). Docs: the treatAsHistorical schema describe (@objectstack/spec, regenerated references/api/export.mdx) now documents all three effects — FSM skip (#3479), audit-timeline preservation (#3493) and undo mirroring (#3556) — instead of only the state-machine half; protocol/objectql/state-machine.mdx gains a bullet on the symmetric undo behavior. Also confirms (via a codebase audit) that the import-undo route was the only business-record snapshot write-back path, so #3556 needs no sibling fix. Claude-Session: https://claude.ai/code/session_01B5rdfBKjkbcoEif4KUV6xE Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4631592 commit c80aece

5 files changed

Lines changed: 166 additions & 3 deletions

File tree

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
---
3+
4+
test+docs: harden and document the historical-import `preserveAudit` seam (#3493 follow-ups)
5+
6+
Follow-ups to #3493 / #3549 / #3556, no behavior change:
7+
8+
- **Real-SQLite end-to-end test** (`@objectstack/runtime`): wires the real ObjectQL
9+
engine to the real `SqlDriver` (better-sqlite3) and proves the `preserveAudit`
10+
seam the mock/in-memory drivers structurally cannot — that `context.preserveAudit`
11+
threads through `buildDriverOptions` and defeats the SQL driver's `updated_at`
12+
force-stamp on both the forward historical write and the undo restore, and that a
13+
business `readonly` field (`closed_at`) survives the engine strip all the way to
14+
disk. Includes the #3549 undo capstone (restore-under-preserveAudit rolls the
15+
timeline back; a plain restore re-stamps now — the bug #3556 fixed).
16+
- **Docs**: the `treatAsHistorical` schema describe (`@objectstack/spec` → regenerated
17+
`references/api/export.mdx`) now documents all three effects — FSM skip (#3479),
18+
audit-timeline preservation (#3493), and undo mirroring (#3556) — instead of only
19+
the state-machine half; and `protocol/objectql/state-machine.mdx` gains a bullet on
20+
the symmetric undo behavior.
21+
22+
Describe-string + regenerated docs + a new test only — releases nothing.

content/docs/protocol/objectql/state-machine.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ transitions: {
102102
- **Seed writes are exempt** (#3433). Curated seed data — package bootstrap fixtures, marketplace templates, per-org replay, all loaded by `SeedLoaderService` — is a snapshot of established facts, not a record walking its lifecycle, so it bypasses the `state_machine` rule entirely: a seed may be born mid-lifecycle (a `completed` project, a `closed_won` opportunity) and neither `initialStates` (insert) nor `transitions` (update) is enforced. Every *other* validation still runs, so a seed must still satisfy field shape, `format`, `script`, and the rest. `os lint` warns when a seeded value is not a state the machine declares, so a typo is still caught before boot.
103103
- **A "historical" data import is exempt too** (#3479). Migrating established facts — a batch of already-`closed` tickets, `closed_won` deals — is the same "snapshot, not a lifecycle event" situation. Set `treatAsHistorical: true` on the import request (default **off**) and the runner puts `skipStateMachine` on the write context, so `initialStates` doesn't reject those mid-lifecycle rows. A normal import leaves it off and still walks the FSM — the strict behavior is the default, so the exemption is always an explicit opt-in.
104104
- **`treatAsHistorical` also preserves the original audit timeline** (#3493). Skipping the FSM is only half of migrating established facts; the other half is keeping *when* they happened and *who* did them. Under the same flag the write context also carries `preserveAudit`, which (1) makes `updated_at` / `updated_by` **client-preferred** — a supplied historical last-modified survives instead of being stamped with the import instant, symmetric with how `created_at` / `created_by` already behave on insert — and (2) admits a **whitelist** through the static-`readonly` write strip: the audit/timestamp family plus author-declared business `readonly` fields (`closed_at`, `resolved_by`, …), so an `upsert` refresh no longer silently drops them. Platform-managed `system` columns outside that family (`organization_id` and other tenancy/generated columns) stay stripped — a historical import reinstates facts, it does not forge tenancy. Like the FSM exemption this is opt-in: a normal write still auto-stamps `updated_at`/`updated_by` and strips `readonly` exactly as before, and permissions / RLS / field-level security are unchanged.
105+
- **Undoing a historical import is symmetric** (#3549 / #3556). The import undo (`POST /data/import/jobs/:jobId/undo`) logically rolls back a finished job — deleting the rows it created and restoring the captured pre-import snapshot on the rows it updated. That restore write now carries `preserveAudit` too, but **only** when the job was flagged `treatAsHistorical`, so the snapshotted `updated_at` / `updated_by` and business `readonly` fields (`closed_at`, …) are reinstated verbatim instead of being re-stamped to the undo instant. Without it the undo would silently overwrite the very timeline the historical import preserved; a normal (non-historical) import's undo keeps the default stamp/strip.
105106

106107
### Conditional transitions
107108

content/docs/references/api/export.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ const result = CreateExportJobRequest.parse(data);
8282
| **writeMode** | `Enum<'insert' \| 'update' \| 'upsert'>` || insert / update / upsert semantics |
8383
| **matchFields** | `string[]` | optional | Fields that identify an existing record (required for update/upsert) |
8484
| **runAutomations** | `boolean` || Fire triggers/hooks for each imported row (off by default for bulk) |
85-
| **treatAsHistorical** | `boolean` || Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM. |
85+
| **treatAsHistorical** | `boolean` || Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556). |
8686
| **trimWhitespace** | `boolean` || Trim leading/trailing whitespace from string cells |
8787
| **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) |
8888
| **createMissingOptions** | `boolean` || Keep unmatched select values instead of failing the row |
@@ -371,7 +371,7 @@ Type: `{ sourceField: string; targetField: string; targetLabel?: string; transfo
371371
| **writeMode** | `Enum<'insert' \| 'update' \| 'upsert'>` || insert / update / upsert semantics |
372372
| **matchFields** | `string[]` | optional | Fields that identify an existing record (required for update/upsert) |
373373
| **runAutomations** | `boolean` || Fire triggers/hooks for each imported row (off by default for bulk) |
374-
| **treatAsHistorical** | `boolean` || Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM. |
374+
| **treatAsHistorical** | `boolean` || Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556). |
375375
| **trimWhitespace** | `boolean` || Trim leading/trailing whitespace from string cells |
376376
| **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) |
377377
| **createMissingOptions** | `boolean` || Keep unmatched select values instead of failing the row |
Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* Real-SQLite end-to-end for the `preserveAudit` seam (#3493 / #3549 / #3556).
5+
*
6+
* A "historical" import (`treatAsHistorical`) writes with `context.preserveAudit`
7+
* so it KEEPS the original `updated_at` and author-declared business `readonly`
8+
* fields (`closed_at`) instead of stamping-now / stripping them. The undo of such
9+
* an import mirrors that flag so restoring the captured pre-import snapshot rolls
10+
* the timeline BACK rather than re-stamping it (#3549 → fixed in #3556).
11+
*
12+
* Each half is already unit-tested in isolation — the engine's audit hook +
13+
* readonly-strip whitelist against a mock driver (objectql `plugin.integration`),
14+
* and the SQL driver's own `updated_at` force-stamp bypass against a directly-
15+
* supplied option (`driver-sql` `sql-driver-timestamp-format`). What NEITHER can
16+
* prove is the SEAM BETWEEN them: that `context.preserveAudit` set on an engine
17+
* write actually threads through `buildDriverOptions` into the driver's options
18+
* and defeats the REAL SQL `updated_at` force-stamp end-to-end. A mock/in-memory
19+
* driver echoes `data.updated_at` and never force-stamps, so it structurally
20+
* cannot catch a break in that thread. This wires the REAL {@link ObjectQL}
21+
* engine to the REAL {@link SqlDriver} (better-sqlite3, on-disk) and reads the
22+
* persisted row back to pin the whole path.
23+
*/
24+
25+
import { describe, it, expect, afterEach } from 'vitest';
26+
import { mkdtempSync, rmSync } from 'node:fs';
27+
import { tmpdir } from 'node:os';
28+
import { join } from 'node:path';
29+
import { ObjectQL } from '@objectstack/objectql';
30+
import { SqlDriver } from '@objectstack/driver-sql';
31+
32+
const ISO_Z = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
33+
const HISTORICAL = '2021-03-01T09:00:00.000Z';
34+
35+
const TICKET = {
36+
name: 'ticket',
37+
fields: {
38+
name: { type: 'text' },
39+
// Author-declared business readonly field — a case's close time. Normally
40+
// stripped on a client write; a historical import (preserveAudit) reinstates it.
41+
closed_at: { type: 'datetime', readonly: true },
42+
},
43+
};
44+
45+
describe('preserveAudit end-to-end on a REAL SqlDriver (#3493 / #3549)', () => {
46+
let engine: ObjectQL | null = null;
47+
let dir: string | null = null;
48+
49+
afterEach(async () => {
50+
try { await engine?.destroy(); } catch { /* noop */ }
51+
engine = null;
52+
if (dir) { rmSync(dir, { recursive: true, force: true }); dir = null; }
53+
});
54+
55+
async function boot() {
56+
dir = mkdtempSync(join(tmpdir(), 'os-preserveaudit-'));
57+
const driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename: join(dir, 'data.sqlite') }, useNullAsDefault: true });
58+
await driver.initObjects([TICKET]); // create the real table + audit columns
59+
engine = new ObjectQL();
60+
engine.registerDriver(driver, true);
61+
await engine.init();
62+
engine.registry.registerObject(TICKET as any);
63+
return engine;
64+
}
65+
66+
const readOne = async (id: string): Promise<any> =>
67+
(await (engine as ObjectQL).find('ticket', { where: { id } }))[0];
68+
69+
it('historical write: preserveAudit keeps the supplied updated_at AND the business readonly closed_at, through the SQL force-stamp', async () => {
70+
const e = await boot();
71+
const rec: any = await e.insert('ticket', { name: 'A' });
72+
const id = rec.id;
73+
74+
await e.update(
75+
'ticket',
76+
{ id, name: 'B', updated_at: HISTORICAL, closed_at: HISTORICAL },
77+
{ context: { preserveAudit: true } } as any,
78+
);
79+
80+
const row = await readOne(id);
81+
expect(row.name).toBe('B');
82+
// The engine threaded preserveAudit into the driver options, so the driver's
83+
// updated_at force-stamp was bypassed and the supplied instant survived to disk.
84+
expect(row.updated_at).toBe(HISTORICAL);
85+
// The engine's readonly-strip whitelist admitted the business readonly field,
86+
// and it was actually written by the real driver (not just echoed by a mock).
87+
expect(row.closed_at).toBe(HISTORICAL);
88+
});
89+
90+
it('normal write (control): without preserveAudit the SQL driver force-stamps updated_at and the engine strips readonly closed_at', async () => {
91+
const e = await boot();
92+
const rec: any = await e.insert('ticket', { name: 'A' });
93+
const id = rec.id;
94+
95+
await e.update(
96+
'ticket',
97+
{ id, name: 'B', updated_at: HISTORICAL, closed_at: HISTORICAL },
98+
{ context: {} } as any,
99+
);
100+
101+
const row = await readOne(id);
102+
expect(row.name).toBe('B');
103+
expect(row.updated_at).toMatch(ISO_Z);
104+
expect(row.updated_at).not.toBe(HISTORICAL); // force-stamped "now", not the supplied instant
105+
expect(row.closed_at ?? null).toBeNull(); // readonly business field stripped — never reached the driver
106+
});
107+
108+
it('undo capstone (#3549): restoring a captured pre-import snapshot under preserveAudit rolls updated_at BACK; without the flag it re-stamps now (the corruption #3556 prevents)', async () => {
109+
const e = await boot();
110+
const rec: any = await e.insert('ticket', { name: 'A' });
111+
const id = rec.id;
112+
const original = await readOne(id);
113+
expect(original.updated_at).toMatch(ISO_Z);
114+
115+
// Ensure "now" is measurably later than the original insert stamp.
116+
await new Promise((r) => setTimeout(r, 5));
117+
118+
// A historical import moves the row's recorded timeline to the historical instant.
119+
await e.update('ticket', { id, name: 'B', updated_at: HISTORICAL }, { context: { preserveAudit: true } } as any);
120+
expect((await readOne(id)).updated_at).toBe(HISTORICAL);
121+
122+
// The pre-import snapshot the undo log captured (payload keys only).
123+
const before = { id, name: original.name, updated_at: original.updated_at };
124+
125+
// Undo WITH preserveAudit (the #3556 fix): the original timeline is restored,
126+
// not re-stamped — the driver's force-stamp is bypassed on the restore write too.
127+
await e.update('ticket', before, { context: { preserveAudit: true, skipAutomations: true } } as any);
128+
expect((await readOne(id)).updated_at).toBe(original.updated_at);
129+
130+
// Control — the SAME restore WITHOUT preserveAudit (the pre-#3556 behavior):
131+
// move the timeline again, then undo with a plain context. The SQL driver
132+
// force-stamps now, so the captured original is silently lost. This is exactly
133+
// the #3549 corruption the fix removes.
134+
await e.update('ticket', { id, name: 'B2', updated_at: HISTORICAL }, { context: { preserveAudit: true } } as any);
135+
await e.update('ticket', before, { context: { skipAutomations: true } } as any);
136+
const corrupted = await readOne(id);
137+
expect(corrupted.updated_at).toMatch(ISO_Z);
138+
expect(corrupted.updated_at).not.toBe(original.updated_at); // timeline NOT restored — the bug shape
139+
});
140+
});

packages/spec/src/api/export.zod.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -320,7 +320,7 @@ export const ImportRequestSchema = lazySchema(() => z.object({
320320
runAutomations: z.boolean().default(false)
321321
.describe('Fire triggers/hooks for each imported row (off by default for bulk)'),
322322
treatAsHistorical: z.boolean().default(false)
323-
.describe('Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM.'),
323+
.describe('Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556).'),
324324
trimWhitespace: z.boolean().default(true)
325325
.describe('Trim leading/trailing whitespace from string cells'),
326326
nullValues: z.array(z.string()).optional()

0 commit comments

Comments
 (0)