From 655ea2cedf9144ec7cee01d2e83b2e336ed5ec5c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 27 Jul 2026 05:06:57 +0000 Subject: [PATCH] test+docs: harden and document the historical-import preserveAudit seam (#3493 follow-ups) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-ups to #3493 / #3549 / #3556 — no behavior change. Real-SQLite end-to-end test (@objectstack/runtime): wires the real ObjectQL engine to the real SqlDriver (better-sqlite3) and proves the preserveAudit seam that mock / in-memory drivers structurally cannot — that context.preserveAudit threads through buildDriverOptions and defeats the SQL driver's updated_at force-stamp on both the forward historical write and the undo restore, and that a business readonly field (closed_at) survives the engine strip all the way to disk. Includes the #3549 undo capstone: restore-under-preserveAudit rolls the timeline back, while a plain restore re-stamps now (the bug #3556 fixed). Docs: the treatAsHistorical schema describe (@objectstack/spec, regenerated references/api/export.mdx) now documents all three effects — FSM skip (#3479), audit-timeline preservation (#3493) and undo mirroring (#3556) — instead of only the state-machine half; protocol/objectql/state-machine.mdx gains a bullet on the symmetric undo behavior. Also confirms (via a codebase audit) that the import-undo route was the only business-record snapshot write-back path, so #3556 needs no sibling fix. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01B5rdfBKjkbcoEif4KUV6xE --- .changeset/preserveaudit-test-and-docs.md | 22 +++ .../docs/protocol/objectql/state-machine.mdx | 1 + content/docs/references/api/export.mdx | 4 +- ...erve-audit-real-driver.integration.test.ts | 140 ++++++++++++++++++ packages/spec/src/api/export.zod.ts | 2 +- 5 files changed, 166 insertions(+), 3 deletions(-) create mode 100644 .changeset/preserveaudit-test-and-docs.md create mode 100644 packages/runtime/src/preserve-audit-real-driver.integration.test.ts diff --git a/.changeset/preserveaudit-test-and-docs.md b/.changeset/preserveaudit-test-and-docs.md new file mode 100644 index 0000000000..7d4a5dc96c --- /dev/null +++ b/.changeset/preserveaudit-test-and-docs.md @@ -0,0 +1,22 @@ +--- +--- + +test+docs: harden and document the historical-import `preserveAudit` seam (#3493 follow-ups) + +Follow-ups to #3493 / #3549 / #3556, no behavior change: + +- **Real-SQLite end-to-end test** (`@objectstack/runtime`): wires the real ObjectQL + engine to the real `SqlDriver` (better-sqlite3) and proves the `preserveAudit` + seam the mock/in-memory drivers structurally cannot — that `context.preserveAudit` + threads through `buildDriverOptions` and defeats the SQL driver's `updated_at` + force-stamp on both the forward historical write and the undo restore, and that a + business `readonly` field (`closed_at`) survives the engine strip all the way to + disk. Includes the #3549 undo capstone (restore-under-preserveAudit rolls the + timeline back; a plain restore re-stamps now — the bug #3556 fixed). +- **Docs**: the `treatAsHistorical` schema describe (`@objectstack/spec` → regenerated + `references/api/export.mdx`) now documents all three effects — FSM skip (#3479), + audit-timeline preservation (#3493), and undo mirroring (#3556) — instead of only + the state-machine half; and `protocol/objectql/state-machine.mdx` gains a bullet on + the symmetric undo behavior. + +Describe-string + regenerated docs + a new test only — releases nothing. diff --git a/content/docs/protocol/objectql/state-machine.mdx b/content/docs/protocol/objectql/state-machine.mdx index d329f0eb12..1784855b6d 100644 --- a/content/docs/protocol/objectql/state-machine.mdx +++ b/content/docs/protocol/objectql/state-machine.mdx @@ -102,6 +102,7 @@ transitions: { - **Seed writes are exempt** (#3433). Curated seed data — package bootstrap fixtures, marketplace templates, per-org replay, all loaded by `SeedLoaderService` — is a snapshot of established facts, not a record walking its lifecycle, so it bypasses the `state_machine` rule entirely: a seed may be born mid-lifecycle (a `completed` project, a `closed_won` opportunity) and neither `initialStates` (insert) nor `transitions` (update) is enforced. Every *other* validation still runs, so a seed must still satisfy field shape, `format`, `script`, and the rest. `os lint` warns when a seeded value is not a state the machine declares, so a typo is still caught before boot. - **A "historical" data import is exempt too** (#3479). Migrating established facts — a batch of already-`closed` tickets, `closed_won` deals — is the same "snapshot, not a lifecycle event" situation. Set `treatAsHistorical: true` on the import request (default **off**) and the runner puts `skipStateMachine` on the write context, so `initialStates` doesn't reject those mid-lifecycle rows. A normal import leaves it off and still walks the FSM — the strict behavior is the default, so the exemption is always an explicit opt-in. - **`treatAsHistorical` also preserves the original audit timeline** (#3493). Skipping the FSM is only half of migrating established facts; the other half is keeping *when* they happened and *who* did them. Under the same flag the write context also carries `preserveAudit`, which (1) makes `updated_at` / `updated_by` **client-preferred** — a supplied historical last-modified survives instead of being stamped with the import instant, symmetric with how `created_at` / `created_by` already behave on insert — and (2) admits a **whitelist** through the static-`readonly` write strip: the audit/timestamp family plus author-declared business `readonly` fields (`closed_at`, `resolved_by`, …), so an `upsert` refresh no longer silently drops them. Platform-managed `system` columns outside that family (`organization_id` and other tenancy/generated columns) stay stripped — a historical import reinstates facts, it does not forge tenancy. Like the FSM exemption this is opt-in: a normal write still auto-stamps `updated_at`/`updated_by` and strips `readonly` exactly as before, and permissions / RLS / field-level security are unchanged. +- **Undoing a historical import is symmetric** (#3549 / #3556). The import undo (`POST /data/import/jobs/:jobId/undo`) logically rolls back a finished job — deleting the rows it created and restoring the captured pre-import snapshot on the rows it updated. That restore write now carries `preserveAudit` too, but **only** when the job was flagged `treatAsHistorical`, so the snapshotted `updated_at` / `updated_by` and business `readonly` fields (`closed_at`, …) are reinstated verbatim instead of being re-stamped to the undo instant. Without it the undo would silently overwrite the very timeline the historical import preserved; a normal (non-historical) import's undo keeps the default stamp/strip. ### Conditional transitions diff --git a/content/docs/references/api/export.mdx b/content/docs/references/api/export.mdx index 8891b52df4..3cb2fb67ae 100644 --- a/content/docs/references/api/export.mdx +++ b/content/docs/references/api/export.mdx @@ -82,7 +82,7 @@ const result = CreateExportJobRequest.parse(data); | **writeMode** | `Enum<'insert' \| 'update' \| 'upsert'>` | ✅ | insert / update / upsert semantics | | **matchFields** | `string[]` | optional | Fields that identify an existing record (required for update/upsert) | | **runAutomations** | `boolean` | ✅ | Fire triggers/hooks for each imported row (off by default for bulk) | -| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM. | +| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556). | | **trimWhitespace** | `boolean` | ✅ | Trim leading/trailing whitespace from string cells | | **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) | | **createMissingOptions** | `boolean` | ✅ | Keep unmatched select values instead of failing the row | @@ -371,7 +371,7 @@ Type: `{ sourceField: string; targetField: string; targetLabel?: string; transfo | **writeMode** | `Enum<'insert' \| 'update' \| 'upsert'>` | ✅ | insert / update / upsert semantics | | **matchFields** | `string[]` | optional | Fields that identify an existing record (required for update/upsert) | | **runAutomations** | `boolean` | ✅ | Fire triggers/hooks for each imported row (off by default for bulk) | -| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM. | +| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556). | | **trimWhitespace** | `boolean` | ✅ | Trim leading/trailing whitespace from string cells | | **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) | | **createMissingOptions** | `boolean` | ✅ | Keep unmatched select values instead of failing the row | diff --git a/packages/runtime/src/preserve-audit-real-driver.integration.test.ts b/packages/runtime/src/preserve-audit-real-driver.integration.test.ts new file mode 100644 index 0000000000..a24e82551f --- /dev/null +++ b/packages/runtime/src/preserve-audit-real-driver.integration.test.ts @@ -0,0 +1,140 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Real-SQLite end-to-end for the `preserveAudit` seam (#3493 / #3549 / #3556). + * + * A "historical" import (`treatAsHistorical`) writes with `context.preserveAudit` + * so it KEEPS the original `updated_at` and author-declared business `readonly` + * fields (`closed_at`) instead of stamping-now / stripping them. The undo of such + * an import mirrors that flag so restoring the captured pre-import snapshot rolls + * the timeline BACK rather than re-stamping it (#3549 → fixed in #3556). + * + * Each half is already unit-tested in isolation — the engine's audit hook + + * readonly-strip whitelist against a mock driver (objectql `plugin.integration`), + * and the SQL driver's own `updated_at` force-stamp bypass against a directly- + * supplied option (`driver-sql` `sql-driver-timestamp-format`). What NEITHER can + * prove is the SEAM BETWEEN them: that `context.preserveAudit` set on an engine + * write actually threads through `buildDriverOptions` into the driver's options + * and defeats the REAL SQL `updated_at` force-stamp end-to-end. A mock/in-memory + * driver echoes `data.updated_at` and never force-stamps, so it structurally + * cannot catch a break in that thread. This wires the REAL {@link ObjectQL} + * engine to the REAL {@link SqlDriver} (better-sqlite3, on-disk) and reads the + * persisted row back to pin the whole path. + */ + +import { describe, it, expect, afterEach } from 'vitest'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; + +const ISO_Z = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/; +const HISTORICAL = '2021-03-01T09:00:00.000Z'; + +const TICKET = { + name: 'ticket', + fields: { + name: { type: 'text' }, + // Author-declared business readonly field — a case's close time. Normally + // stripped on a client write; a historical import (preserveAudit) reinstates it. + closed_at: { type: 'datetime', readonly: true }, + }, +}; + +describe('preserveAudit end-to-end on a REAL SqlDriver (#3493 / #3549)', () => { + let engine: ObjectQL | null = null; + let dir: string | null = null; + + afterEach(async () => { + try { await engine?.destroy(); } catch { /* noop */ } + engine = null; + if (dir) { rmSync(dir, { recursive: true, force: true }); dir = null; } + }); + + async function boot() { + dir = mkdtempSync(join(tmpdir(), 'os-preserveaudit-')); + const driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename: join(dir, 'data.sqlite') }, useNullAsDefault: true }); + await driver.initObjects([TICKET]); // create the real table + audit columns + engine = new ObjectQL(); + engine.registerDriver(driver, true); + await engine.init(); + engine.registry.registerObject(TICKET as any); + return engine; + } + + const readOne = async (id: string): Promise => + (await (engine as ObjectQL).find('ticket', { where: { id } }))[0]; + + it('historical write: preserveAudit keeps the supplied updated_at AND the business readonly closed_at, through the SQL force-stamp', async () => { + const e = await boot(); + const rec: any = await e.insert('ticket', { name: 'A' }); + const id = rec.id; + + await e.update( + 'ticket', + { id, name: 'B', updated_at: HISTORICAL, closed_at: HISTORICAL }, + { context: { preserveAudit: true } } as any, + ); + + const row = await readOne(id); + expect(row.name).toBe('B'); + // The engine threaded preserveAudit into the driver options, so the driver's + // updated_at force-stamp was bypassed and the supplied instant survived to disk. + expect(row.updated_at).toBe(HISTORICAL); + // The engine's readonly-strip whitelist admitted the business readonly field, + // and it was actually written by the real driver (not just echoed by a mock). + expect(row.closed_at).toBe(HISTORICAL); + }); + + it('normal write (control): without preserveAudit the SQL driver force-stamps updated_at and the engine strips readonly closed_at', async () => { + const e = await boot(); + const rec: any = await e.insert('ticket', { name: 'A' }); + const id = rec.id; + + await e.update( + 'ticket', + { id, name: 'B', updated_at: HISTORICAL, closed_at: HISTORICAL }, + { context: {} } as any, + ); + + const row = await readOne(id); + expect(row.name).toBe('B'); + expect(row.updated_at).toMatch(ISO_Z); + expect(row.updated_at).not.toBe(HISTORICAL); // force-stamped "now", not the supplied instant + expect(row.closed_at ?? null).toBeNull(); // readonly business field stripped — never reached the driver + }); + + it('undo capstone (#3549): restoring a captured pre-import snapshot under preserveAudit rolls updated_at BACK; without the flag it re-stamps now (the corruption #3556 prevents)', async () => { + const e = await boot(); + const rec: any = await e.insert('ticket', { name: 'A' }); + const id = rec.id; + const original = await readOne(id); + expect(original.updated_at).toMatch(ISO_Z); + + // Ensure "now" is measurably later than the original insert stamp. + await new Promise((r) => setTimeout(r, 5)); + + // A historical import moves the row's recorded timeline to the historical instant. + await e.update('ticket', { id, name: 'B', updated_at: HISTORICAL }, { context: { preserveAudit: true } } as any); + expect((await readOne(id)).updated_at).toBe(HISTORICAL); + + // The pre-import snapshot the undo log captured (payload keys only). + const before = { id, name: original.name, updated_at: original.updated_at }; + + // Undo WITH preserveAudit (the #3556 fix): the original timeline is restored, + // not re-stamped — the driver's force-stamp is bypassed on the restore write too. + await e.update('ticket', before, { context: { preserveAudit: true, skipAutomations: true } } as any); + expect((await readOne(id)).updated_at).toBe(original.updated_at); + + // Control — the SAME restore WITHOUT preserveAudit (the pre-#3556 behavior): + // move the timeline again, then undo with a plain context. The SQL driver + // force-stamps now, so the captured original is silently lost. This is exactly + // the #3549 corruption the fix removes. + await e.update('ticket', { id, name: 'B2', updated_at: HISTORICAL }, { context: { preserveAudit: true } } as any); + await e.update('ticket', before, { context: { skipAutomations: true } } as any); + const corrupted = await readOne(id); + expect(corrupted.updated_at).toMatch(ISO_Z); + expect(corrupted.updated_at).not.toBe(original.updated_at); // timeline NOT restored — the bug shape + }); +}); diff --git a/packages/spec/src/api/export.zod.ts b/packages/spec/src/api/export.zod.ts index c913f24a04..7d6d4e2673 100644 --- a/packages/spec/src/api/export.zod.ts +++ b/packages/spec/src/api/export.zod.ts @@ -320,7 +320,7 @@ export const ImportRequestSchema = lazySchema(() => z.object({ runAutomations: z.boolean().default(false) .describe('Fire triggers/hooks for each imported row (off by default for bulk)'), treatAsHistorical: z.boolean().default(false) - .describe('Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM.'), + .describe('Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556).'), trimWhitespace: z.boolean().default(true) .describe('Trim leading/trailing whitespace from string cells'), nullValues: z.array(z.string()).optional()