feat(migrate): a datastore created from empty attests its data migrations at creation (#3438) (#4215)

* feat(migrate): a datastore created from empty attests its data migrations at creation (#3438)

ADR-0104's 2026-07-30 addendum: deployment-level migration flags could only be
recorded by a migration run, so a database created on a version that already
ships the migrations started lax and stayed lax until someone ran a command
that, for them, converts nothing. Every new deployment re-entered the warn
regime — and since #3459, kept every released file forever.

A store the platform creates from empty now records both flags at that moment.
The fact is observed, not assumed, and the test is deliberately strict: every
table made by this boot and none found already present. One pre-existing table,
one datasource that was already there, one driver that cannot account for its
sync — any of those and the deployment attests nothing.

Neither seam the addendum suggested survived: "sys_migration is absent" is the
16-to-17 trap (an upgrading database lacks it too) and the schema bootstrap runs
identically on both. Nothing recorded datastore newness at all — the drivers
knew it for one statement and discarded it — so the observation is now kept as
created-vs-found table counts (IDataDriver.getSchemaSyncStats, SQL + memory),
aggregated by the engine and consumed at kernel:ready by the service that owns
sys_migration. Counted per table, not per sync call: boot syncs more than once,
and treating a re-sync of our own table as history would disable attestation
everywhere.

Attestation never overwrites an existing row and never throws into a boot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KM46HedAWLbkuZyMBXn5wJ

* test(driver-sql): close both pools unconditionally in the schema-sync-stats test

A knex pool outlives a failed assertion with a live timer, which keeps the
test process alive long after the run itself has passed — the shape of failure
that shows up as "every test passed, the job never exited".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KM46HedAWLbkuZyMBXn5wJ

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/adr-0104-attestation-adr-note.md

@@ -0,0 +1,4 @@
+---
+---
+
+docs(adr-0104): record the fresh-datastore attestation seam as implemented — driver-reported created-vs-found table counts, not "sys_migration is missing" (which is the 16→17 trap) — and note that born-attested dogfood now exercises the #3459 collection path on every boot. Releases nothing.

.changeset/adr-0104-fresh-datastore-attestation.md

@@ -0,0 +1,47 @@
+---
+"@objectstack/spec": minor
+"@objectstack/platform-objects": minor
+"@objectstack/objectql": minor
+"@objectstack/driver-memory": minor
+"@objectstack/driver-sql": minor
+"@objectstack/service-storage": minor
+---
+
+feat(migrate): a datastore created from empty attests its data migrations at creation (#3438, ADR-0104 2026-07-30 addendum)
+
+Deployment-level migration flags could only be recorded by running
+`os migrate`. That left a hole at the other end of a deployment's life: a
+database created on a version that already ships the migrations started **lax**
+and stayed lax until someone thought to run a command that, for them, converts
+nothing and finds nothing. Every new deployment re-entered the warn regime, so
+the warn regime would never die out — and, since #3459, every new deployment
+also kept every released file forever.
+
+A store the platform **creates from empty** now records
+`adr-0104-file-references` and `adr-0104-value-shapes` at that moment. Nothing
+to run; enforcement and collection are live from the first boot.
+
+**This is not version-gating in disguise.** The fact recorded — no legacy value
+is stored here — is *observed*: the store had no history at all. The platform
+attests only what it watched itself create, and the test is deliberately
+strict: every table made by this boot and **none found already present**. One
+pre-existing table anywhere, one datasource that was already there, one driver
+that cannot account for its schema sync — any of those and the deployment
+attests nothing and produces its evidence by scan, exactly as before. "Found
+empty" and "created empty" are not the same claim, and only the second is an
+observation.
+
+**New surfaces.** `IDataDriver.getSchemaSyncStats?()` (optional, purely
+observational: tables created vs found since connect — implemented by the SQL
+and in-memory drivers), `engine.wasDatastoreCreatedFromEmpty()`,
+`attestFreshDatastore()` in `@objectstack/platform-objects/system`, and
+`VALUE_SHAPES_MIGRATION_ID` / `CREATION_ATTESTED_MIGRATION_IDS` in
+`@objectstack/spec/system`. Attestation never overwrites an existing flag row
+and never throws into a boot: a failure leaves the deployment lax, which a
+migration run can still fix.
+
+**Upgrading changes nothing for an existing database.** It is non-empty when
+the platform reaches it, so it is never attested — run
+`os migrate files-to-references --apply` as before. Importing legacy values
+into an attested deployment is rejected loudly at the write path;
+`OS_ALLOW_LAX_MEDIA_VALUES=1` re-opens leniency while you diagnose.

content/docs/deployment/cli.mdx

@@ -675,6 +675,25 @@ delete check re-reads the flag fresh, so a later failing run stops collection
 without a restart.
 </Callout>
 
+#### A database created by this version needs no migration
+
+A deployment whose database the platform **creates from empty** records these
+flags at that moment, so it is enforcing from its first boot and never enters
+the warn regime at all. Nothing to run: the fact a migration would establish —
+no legacy value is stored here — is already settled by the store having no
+history.
+
+The platform attests this only for a store it watched itself create: every
+table made by that first boot, none found already present. A database that
+existed before — an upgrade, a restore, a store shared with anything else —
+attests nothing and produces its evidence by running the command, because
+"found empty" and "created empty" are not the same claim.
+
+Importing legacy values into such a deployment is rejected at the write path
+rather than silently accepted. That is the intended outcome; if you must admit
+them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` re-opens leniency, and
+re-running the migration re-establishes the flag from the data itself.
+
 ### Scaffolding
 
 | Command | Alias | Description |

content/docs/releases/v17.mdx

@@ -984,6 +984,13 @@ records** — never the platform version — is what authorises both strict medi
 value shapes and irreversible file collection. Upgrading changes neither;
 running the migration does.
 
+**A database created by 17 attests both flags at creation** (#3438), so a new
+deployment enforces from its first boot instead of waiting for someone to run a
+migration that, for an empty store, does nothing. The platform attests only a
+store it watched itself create — every table made by that boot, none found
+already there; an upgraded or restored database attests nothing and produces its
+evidence by running the command.
+
 **Released-file collection is live behind that same flag** (#3459). On a
 verified deployment, a field file whose one owning record lets go — the field
 cleared, or the record deleted — is tombstoned into the declared 30-day grace

docs/adr/0104-field-runtime-value-shape-contract.md

@@ -855,10 +855,29 @@ implementing PR's decision (candidates: the system seed that first creates
 requirement: it fires only for a store it is itself creating from empty,
 never for one it found.
 
+**Seam, as implemented.** Neither candidate survived contact: *"`sys_migration`
+is absent"* is precisely the 16→17 trap — an upgrading database lacks that
+table too — and the schema bootstrap runs identically on both. Nothing in the
+platform recorded datastore newness at all; the drivers knew it for one
+statement (`hasTable` → `createTable`) and discarded it. So the observation is
+now kept: each driver counts the tables it **created** against those it
+**found** since connect (`IDataDriver.getSchemaSyncStats`), and a store is
+attested only when the whole engine reports *created > 0 and existing = 0*.
+Every uncertainty resolves to "no" — a driver that cannot report, a deferred or
+skipped sync, a second datasource that was already there. Being strict here is
+cheap in the only direction that matters: a wrongly-withheld attestation costs
+a command someone can still run, a wrongly-granted one asserts a store's
+history on no evidence. The write lands at `kernel:ready` from the service that
+owns `sys_migration`, and never overwrites an existing row: a store with flag
+rows is by definition not one being created.
+
 Born-strict deployments also make the platform's own dogfood the standing
 canary for R2 (a codified shape stricter than some legitimate client's
 writes): showcase/CRM boots are fresh datastores, so they enforce from birth,
-and a false rejection fails our suites before any customer sees it.
+and a false rejection fails our suites before any customer sees it. Since
+#3459 that canary covers the collection path too — a born-attested dogfood run
+exercises release-time tombstoning and the reap guard's re-verify on every
+boot, so an over-eager delete fails our suites rather than a customer's data.
 
 ### D2 action params: the evidence cannot exist, so strict is the 17.0 default
 

packages/objectql/src/engine.test.ts

@@ -2274,4 +2274,48 @@ describe('ObjectQL — file-as-reference migration flag (#3617)', () => {
     const flagReads = vi.mocked(driver.find).mock.calls.filter((c) => c[0] === 'sys_migration');
     expect(flagReads).toHaveLength(2);
   });
+
+  // ── Was this datastore created from empty? (#3438, ADR-0104) ──────
+  // The one input to fresh-datastore attestation. Permission is granted on
+  // this answer, so every uncertainty must resolve to `false`.
+  describe('wasDatastoreCreatedFromEmpty', () => {
+    const withStats = (d: IDataDriver, stats: { created: number; existing: number } | null) => {
+      if (stats) (d as any).getSchemaSyncStats = () => stats;
+      else delete (d as any).getSchemaSyncStats;
+    };
+
+    it('is true when the driver created tables and found none', () => {
+      withStats(driver, { created: 12, existing: 0 });
+      expect(engine.wasDatastoreCreatedFromEmpty()).toBe(true);
+    });
+
+    it('is false when even one table was already there', () => {
+      withStats(driver, { created: 11, existing: 1 });
+      expect(engine.wasDatastoreCreatedFromEmpty()).toBe(false);
+    });
+
+    it('is false when nothing was created (sync skipped or deferred)', () => {
+      withStats(driver, { created: 0, existing: 0 });
+      expect(engine.wasDatastoreCreatedFromEmpty()).toBe(false);
+    });
+
+    it('is false when no driver can account for its schema sync', () => {
+      withStats(driver, null);
+      expect(engine.wasDatastoreCreatedFromEmpty()).toBe(false);
+    });
+
+    it('is false when a second datasource was not created by this boot', () => {
+      withStats(driver, { created: 12, existing: 0 });
+      const other = {
+        name: 'other',
+        connect: vi.fn().mockResolvedValue(undefined),
+        disconnect: vi.fn().mockResolvedValue(undefined),
+        capabilities: {} as any,
+      } as unknown as IDataDriver;
+      withStats(other, { created: 0, existing: 5 });
+      engine.registerDriver(other);
+
+      expect(engine.wasDatastoreCreatedFromEmpty()).toBe(false);
+    });
+  });
 });

packages/objectql/src/engine.ts

@@ -2262,6 +2262,36 @@ export class ObjectQL implements IDataEngine {
     this.fileReferencesMigrationVerified = null;
   }
 
+  /**
+   * Did this process CREATE the datastore it is talking to, from empty?
+   *
+   * True only when every driver that can account for its schema sync created
+   * tables and found none already there. That conjunction is the whole point:
+   * one pre-existing table anywhere means something ran here before us, so
+   * this store's history is not ours to vouch for. A driver that cannot
+   * report (`getSchemaSyncStats` absent, deferred DDL, sync skipped) makes the
+   * answer no rather than maybe — the consumers of this fact grant
+   * permissions, so "cannot say" must read as "no".
+   *
+   * The one caller is the fresh-datastore attestation (#3438, ADR-0104's
+   * 2026-07-30 addendum), which records that a store born empty needs no data
+   * migration. Ask it only during boot: once the process starts serving, the
+   * counts describe a moment that has passed.
+   */
+  wasDatastoreCreatedFromEmpty(): boolean {
+    let created = 0;
+    let existing = 0;
+    let reporting = 0;
+    for (const driver of this.drivers.values()) {
+      const stats = (driver as any).getSchemaSyncStats?.();
+      if (!stats) continue;
+      reporting += 1;
+      created += Number(stats.created) || 0;
+      existing += Number(stats.existing) || 0;
+    }
+    return reporting > 0 && existing === 0 && created > 0;
+  }
+
   async destroy() {
     this.logger.info('Destroying ObjectQL engine', { driverCount: this.drivers.size });
 

packages/platform-objects/src/system/index.ts

@@ -17,6 +17,9 @@ export {
   readDataMigrationFlag,
   isDataMigrationVerified,
   recordDataMigrationRun,
+  attestFreshDatastore,
+  CREATION_ATTESTATION_DETAIL,
   type MigrationFlagEngine,
   type DataMigrationRunOutcome,
+  type AttestationLogger,
 } from './migration-flag.js';

packages/platform-objects/src/system/migration-flag.test.ts

@@ -1,10 +1,13 @@
 // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
 
-import { describe, it, expect } from 'vitest';
+import { describe, it, expect, vi } from 'vitest';
+import { CREATION_ATTESTED_MIGRATION_IDS } from '@objectstack/spec/system';
 import {
   readDataMigrationFlag,
   isDataMigrationVerified,
   recordDataMigrationRun,
+  attestFreshDatastore,
+  CREATION_ATTESTATION_DETAIL,
   type MigrationFlagEngine,
 } from './migration-flag.js';
 
@@ -112,3 +115,85 @@ describe('deployment-level data-migration flags (#3617)', () => {
     expect(await isDataMigrationVerified(engine, MIGRATION)).toBe(false);
   });
 });
+
+describe('fresh-datastore attestation (ADR-0104, 2026-07-30 addendum)', () => {
+  it('attests every creation-attested migration, verified and blocking-free', async () => {
+    const engine = fakeEngine();
+
+    const attested = await attestFreshDatastore(engine);
+
+    expect(attested).toEqual([...CREATION_ATTESTED_MIGRATION_IDS]);
+    expect(engine.tables.sys_migration).toHaveLength(CREATION_ATTESTED_MIGRATION_IDS.length);
+    for (const id of CREATION_ATTESTED_MIGRATION_IDS) {
+      expect(await isDataMigrationVerified(engine, id)).toBe(true);
+    }
+  });
+
+  it('records nothing as applied — no backfill ran, and none was needed', async () => {
+    const engine = fakeEngine();
+
+    await attestFreshDatastore(engine);
+
+    const row = engine.tables.sys_migration[0];
+    expect(row.applied_at).toBeNull();
+    expect(row.blocking).toBe(0);
+  });
+
+  it('marks the row so evidence-by-birth is distinguishable from evidence-by-scan', async () => {
+    const engine = fakeEngine();
+
+    await attestFreshDatastore(engine);
+
+    expect(JSON.parse(String(engine.tables.sys_migration[0].details))).toEqual(
+      CREATION_ATTESTATION_DETAIL,
+    );
+  });
+
+  /**
+   * The load-bearing safety property. An existing row means this store is not
+   * one being created — whatever the caller believed — so attestation must
+   * leave it exactly as it found it. Overwriting could only ever RAISE a gate
+   * the deployment's own evidence had closed.
+   */
+  it('never overwrites an existing row, including one a failed run closed', async () => {
+    const engine = fakeEngine([
+      { id: MIGRATION, last_run_at: 'yesterday', verified_at: null, blocking: 7 },
+    ]);
+
+    const attested = await attestFreshDatastore(engine);
+
+    expect(attested).not.toContain(MIGRATION);
+    const row = engine.tables.sys_migration.find((r) => r.id === MIGRATION)!;
+    expect(row).toMatchObject({ verified_at: null, blocking: 7, last_run_at: 'yesterday' });
+    expect(await isDataMigrationVerified(engine, MIGRATION)).toBe(false);
+  });
+
+  it('is idempotent — a second call adds nothing', async () => {
+    const engine = fakeEngine();
+
+    await attestFreshDatastore(engine);
+    const second = await attestFreshDatastore(engine);
+
+    expect(second).toEqual([]);
+    expect(engine.tables.sys_migration).toHaveLength(CREATION_ATTESTED_MIGRATION_IDS.length);
+  });
+
+  it('does nothing when sys_migration is not registered (bare kernel)', async () => {
+    const engine = fakeEngine([], { registered: false });
+
+    expect(await attestFreshDatastore(engine)).toEqual([]);
+    expect(engine.tables.sys_migration).toHaveLength(0);
+  });
+
+  it('a write failure warns and leaves the deployment lax — it never throws into a boot', async () => {
+    const engine = fakeEngine();
+    engine.insert = async () => {
+      throw new Error('table locked');
+    };
+    const logger = { info: vi.fn(), warn: vi.fn() };
+
+    await expect(attestFreshDatastore(engine, { logger })).resolves.toEqual([]);
+    expect(logger.warn).toHaveBeenCalled();
+    expect(logger.info).not.toHaveBeenCalled();
+  });
+});