test+docs: harden and document the historical-import preserveAudit seam (#3493 follow-ups)

.changeset/preserveaudit-test-and-docs.md

@@ -0,0 +1,22 @@
+---
+---
+
+test+docs: harden and document the historical-import `preserveAudit` seam (#3493 follow-ups)
+
+Follow-ups to #3493 / #3549 / #3556, no behavior change:
+
+- **Real-SQLite end-to-end test** (`@objectstack/runtime`): wires the real ObjectQL
+  engine to the real `SqlDriver` (better-sqlite3) and proves the `preserveAudit`
+  seam the mock/in-memory drivers structurally cannot — that `context.preserveAudit`
+  threads through `buildDriverOptions` and defeats the SQL driver's `updated_at`
+  force-stamp on both the forward historical write and the undo restore, and that a
+  business `readonly` field (`closed_at`) survives the engine strip all the way to
+  disk. Includes the #3549 undo capstone (restore-under-preserveAudit rolls the
+  timeline back; a plain restore re-stamps now — the bug #3556 fixed).
+- **Docs**: the `treatAsHistorical` schema describe (`@objectstack/spec` → regenerated
+  `references/api/export.mdx`) now documents all three effects — FSM skip (#3479),
+  audit-timeline preservation (#3493), and undo mirroring (#3556) — instead of only
+  the state-machine half; and `protocol/objectql/state-machine.mdx` gains a bullet on
+  the symmetric undo behavior.
+
+Describe-string + regenerated docs + a new test only — releases nothing.

content/docs/protocol/objectql/state-machine.mdx

@@ -102,6 +102,7 @@ transitions: {
 - **Seed writes are exempt** (#3433). Curated seed data — package bootstrap fixtures, marketplace templates, per-org replay, all loaded by `SeedLoaderService` — is a snapshot of established facts, not a record walking its lifecycle, so it bypasses the `state_machine` rule entirely: a seed may be born mid-lifecycle (a `completed` project, a `closed_won` opportunity) and neither `initialStates` (insert) nor `transitions` (update) is enforced. Every *other* validation still runs, so a seed must still satisfy field shape, `format`, `script`, and the rest. `os lint` warns when a seeded value is not a state the machine declares, so a typo is still caught before boot.
 - **A "historical" data import is exempt too** (#3479). Migrating established facts — a batch of already-`closed` tickets, `closed_won` deals — is the same "snapshot, not a lifecycle event" situation. Set `treatAsHistorical: true` on the import request (default **off**) and the runner puts `skipStateMachine` on the write context, so `initialStates` doesn't reject those mid-lifecycle rows. A normal import leaves it off and still walks the FSM — the strict behavior is the default, so the exemption is always an explicit opt-in.
 - **`treatAsHistorical` also preserves the original audit timeline** (#3493). Skipping the FSM is only half of migrating established facts; the other half is keeping *when* they happened and *who* did them. Under the same flag the write context also carries `preserveAudit`, which (1) makes `updated_at` / `updated_by` **client-preferred** — a supplied historical last-modified survives instead of being stamped with the import instant, symmetric with how `created_at` / `created_by` already behave on insert — and (2) admits a **whitelist** through the static-`readonly` write strip: the audit/timestamp family plus author-declared business `readonly` fields (`closed_at`, `resolved_by`, …), so an `upsert` refresh no longer silently drops them. Platform-managed `system` columns outside that family (`organization_id` and other tenancy/generated columns) stay stripped — a historical import reinstates facts, it does not forge tenancy. Like the FSM exemption this is opt-in: a normal write still auto-stamps `updated_at`/`updated_by` and strips `readonly` exactly as before, and permissions / RLS / field-level security are unchanged.
+- **Undoing a historical import is symmetric** (#3549 / #3556). The import undo (`POST /data/import/jobs/:jobId/undo`) logically rolls back a finished job — deleting the rows it created and restoring the captured pre-import snapshot on the rows it updated. That restore write now carries `preserveAudit` too, but **only** when the job was flagged `treatAsHistorical`, so the snapshotted `updated_at` / `updated_by` and business `readonly` fields (`closed_at`, …) are reinstated verbatim instead of being re-stamped to the undo instant. Without it the undo would silently overwrite the very timeline the historical import preserved; a normal (non-historical) import's undo keeps the default stamp/strip.
 
 ### Conditional transitions
 

content/docs/references/api/export.mdx

@@ -82,7 +82,7 @@ const result = CreateExportJobRequest.parse(data);
 | **writeMode** | `Enum<'insert' \| 'update' \| 'upsert'>` | ✅ | insert / update / upsert semantics |
 | **matchFields** | `string[]` | optional | Fields that identify an existing record (required for update/upsert) |
 | **runAutomations** | `boolean` | ✅ | Fire triggers/hooks for each imported row (off by default for bulk) |
-| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM. |
+| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556). |
 | **trimWhitespace** | `boolean` | ✅ | Trim leading/trailing whitespace from string cells |
 | **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) |
 | **createMissingOptions** | `boolean` | ✅ | Keep unmatched select values instead of failing the row |
@@ -371,7 +371,7 @@ Type: `{ sourceField: string; targetField: string; targetLabel?: string; transfo
 | **writeMode** | `Enum<'insert' \| 'update' \| 'upsert'>` | ✅ | insert / update / upsert semantics |
 | **matchFields** | `string[]` | optional | Fields that identify an existing record (required for update/upsert) |
 | **runAutomations** | `boolean` | ✅ | Fire triggers/hooks for each imported row (off by default for bulk) |
-| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM. |
+| **treatAsHistorical** | `boolean` | ✅ | Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556). |
 | **trimWhitespace** | `boolean` | ✅ | Trim leading/trailing whitespace from string cells |
 | **nullValues** | `string[]` | optional | Strings treated as null/blank (besides empty string) |
 | **createMissingOptions** | `boolean` | ✅ | Keep unmatched select values instead of failing the row |

packages/runtime/src/preserve-audit-real-driver.integration.test.ts

@@ -0,0 +1,140 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * Real-SQLite end-to-end for the `preserveAudit` seam (#3493 / #3549 / #3556).
+ *
+ * A "historical" import (`treatAsHistorical`) writes with `context.preserveAudit`
+ * so it KEEPS the original `updated_at` and author-declared business `readonly`
+ * fields (`closed_at`) instead of stamping-now / stripping them. The undo of such
+ * an import mirrors that flag so restoring the captured pre-import snapshot rolls
+ * the timeline BACK rather than re-stamping it (#3549 → fixed in #3556).
+ *
+ * Each half is already unit-tested in isolation — the engine's audit hook +
+ * readonly-strip whitelist against a mock driver (objectql `plugin.integration`),
+ * and the SQL driver's own `updated_at` force-stamp bypass against a directly-
+ * supplied option (`driver-sql` `sql-driver-timestamp-format`). What NEITHER can
+ * prove is the SEAM BETWEEN them: that `context.preserveAudit` set on an engine
+ * write actually threads through `buildDriverOptions` into the driver's options
+ * and defeats the REAL SQL `updated_at` force-stamp end-to-end. A mock/in-memory
+ * driver echoes `data.updated_at` and never force-stamps, so it structurally
+ * cannot catch a break in that thread. This wires the REAL {@link ObjectQL}
+ * engine to the REAL {@link SqlDriver} (better-sqlite3, on-disk) and reads the
+ * persisted row back to pin the whole path.
+ */
+
+import { describe, it, expect, afterEach } from 'vitest';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { ObjectQL } from '@objectstack/objectql';
+import { SqlDriver } from '@objectstack/driver-sql';
+
+const ISO_Z = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
+const HISTORICAL = '2021-03-01T09:00:00.000Z';
+
+const TICKET = {
+  name: 'ticket',
+  fields: {
+    name: { type: 'text' },
+    // Author-declared business readonly field — a case's close time. Normally
+    // stripped on a client write; a historical import (preserveAudit) reinstates it.
+    closed_at: { type: 'datetime', readonly: true },
+  },
+};
+
+describe('preserveAudit end-to-end on a REAL SqlDriver (#3493 / #3549)', () => {
+  let engine: ObjectQL | null = null;
+  let dir: string | null = null;
+
+  afterEach(async () => {
+    try { await engine?.destroy(); } catch { /* noop */ }
+    engine = null;
+    if (dir) { rmSync(dir, { recursive: true, force: true }); dir = null; }
+  });
+
+  async function boot() {
+    dir = mkdtempSync(join(tmpdir(), 'os-preserveaudit-'));
+    const driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename: join(dir, 'data.sqlite') }, useNullAsDefault: true });
+    await driver.initObjects([TICKET]); // create the real table + audit columns
+    engine = new ObjectQL();
+    engine.registerDriver(driver, true);
+    await engine.init();
+    engine.registry.registerObject(TICKET as any);
+    return engine;
+  }
+
+  const readOne = async (id: string): Promise<any> =>
+    (await (engine as ObjectQL).find('ticket', { where: { id } }))[0];
+
+  it('historical write: preserveAudit keeps the supplied updated_at AND the business readonly closed_at, through the SQL force-stamp', async () => {
+    const e = await boot();
+    const rec: any = await e.insert('ticket', { name: 'A' });
+    const id = rec.id;
+
+    await e.update(
+      'ticket',
+      { id, name: 'B', updated_at: HISTORICAL, closed_at: HISTORICAL },
+      { context: { preserveAudit: true } } as any,
+    );
+
+    const row = await readOne(id);
+    expect(row.name).toBe('B');
+    // The engine threaded preserveAudit into the driver options, so the driver's
+    // updated_at force-stamp was bypassed and the supplied instant survived to disk.
+    expect(row.updated_at).toBe(HISTORICAL);
+    // The engine's readonly-strip whitelist admitted the business readonly field,
+    // and it was actually written by the real driver (not just echoed by a mock).
+    expect(row.closed_at).toBe(HISTORICAL);
+  });
+
+  it('normal write (control): without preserveAudit the SQL driver force-stamps updated_at and the engine strips readonly closed_at', async () => {
+    const e = await boot();
+    const rec: any = await e.insert('ticket', { name: 'A' });
+    const id = rec.id;
+
+    await e.update(
+      'ticket',
+      { id, name: 'B', updated_at: HISTORICAL, closed_at: HISTORICAL },
+      { context: {} } as any,
+    );
+
+    const row = await readOne(id);
+    expect(row.name).toBe('B');
+    expect(row.updated_at).toMatch(ISO_Z);
+    expect(row.updated_at).not.toBe(HISTORICAL); // force-stamped "now", not the supplied instant
+    expect(row.closed_at ?? null).toBeNull();     // readonly business field stripped — never reached the driver
+  });
+
+  it('undo capstone (#3549): restoring a captured pre-import snapshot under preserveAudit rolls updated_at BACK; without the flag it re-stamps now (the corruption #3556 prevents)', async () => {
+    const e = await boot();
+    const rec: any = await e.insert('ticket', { name: 'A' });
+    const id = rec.id;
+    const original = await readOne(id);
+    expect(original.updated_at).toMatch(ISO_Z);
+
+    // Ensure "now" is measurably later than the original insert stamp.
+    await new Promise((r) => setTimeout(r, 5));
+
+    // A historical import moves the row's recorded timeline to the historical instant.
+    await e.update('ticket', { id, name: 'B', updated_at: HISTORICAL }, { context: { preserveAudit: true } } as any);
+    expect((await readOne(id)).updated_at).toBe(HISTORICAL);
+
+    // The pre-import snapshot the undo log captured (payload keys only).
+    const before = { id, name: original.name, updated_at: original.updated_at };
+
+    // Undo WITH preserveAudit (the #3556 fix): the original timeline is restored,
+    // not re-stamped — the driver's force-stamp is bypassed on the restore write too.
+    await e.update('ticket', before, { context: { preserveAudit: true, skipAutomations: true } } as any);
+    expect((await readOne(id)).updated_at).toBe(original.updated_at);
+
+    // Control — the SAME restore WITHOUT preserveAudit (the pre-#3556 behavior):
+    // move the timeline again, then undo with a plain context. The SQL driver
+    // force-stamps now, so the captured original is silently lost. This is exactly
+    // the #3549 corruption the fix removes.
+    await e.update('ticket', { id, name: 'B2', updated_at: HISTORICAL }, { context: { preserveAudit: true } } as any);
+    await e.update('ticket', before, { context: { skipAutomations: true } } as any);
+    const corrupted = await readOne(id);
+    expect(corrupted.updated_at).toMatch(ISO_Z);
+    expect(corrupted.updated_at).not.toBe(original.updated_at); // timeline NOT restored — the bug shape
+  });
+});

packages/spec/src/api/export.zod.ts

@@ -320,7 +320,7 @@ export const ImportRequestSchema = lazySchema(() => z.object({
   runAutomations: z.boolean().default(false)
     .describe('Fire triggers/hooks for each imported row (off by default for bulk)'),
   treatAsHistorical: z.boolean().default(false)
-    .describe('Import as established historical facts: skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479). Off by default so a normal import still walks the FSM.'),
+    .describe('Import as established historical facts. Two effects, both off by default so a normal import is unchanged: (1) skip the state_machine rule so mid-lifecycle rows (e.g. already-closed tickets, closed_won deals) are not rejected by initialStates (#3479); and (2) preserve the original audit timeline — keep the supplied created_at / updated_at / updated_by and author-declared business readonly fields (e.g. closed_at, resolved_by) instead of stamping-now / stripping them (#3493). Undoing a historical import mirrors (2): the captured pre-import values are restored verbatim rather than re-stamped (#3556).'),
   trimWhitespace: z.boolean().default(true)
     .describe('Trim leading/trailing whitespace from string cells'),
   nullValues: z.array(z.string()).optional()