fix(migrate): advertise the value-shape gate, and stop counting fields nothing enforces (#3438)

The scan shipped with no way to find out it exists. ADR-0104's rollout asked
for two advertisements and neither was built: `os migrate meta` — the command
a 16→17 upgrade already runs — never mentioned the data migrations, and a
warn-first deployment booted silently. The command landed and the posture it
governs never changed, because a gate nobody is told about is served by nobody.

`os migrate meta` now closes by naming both gated data migrations with what
each covers and the gate it records, on the human path and in `--json` alike,
including the run that rewrote nothing — an upgrade needing no source change is
exactly where "already done" is easy to assume. A kernel booting with covered
fields and no verified `adr-0104-value-shapes` row logs one line naming the
command that ends warn mode; every case where that line would be false or
useless (an env switch already settled the posture, no covered field is
registered, the gate is already open) answers null and stays silent.

Deliberately WITHOUT reading gate status from the datastore inside `os migrate
meta`, which the ADR's sketch of this advisory assumed it would show. That
command rewrites metadata SOURCE and has never opened a database — its e2e test
spawns the real CLI against a project in a tmpdir. Booting one to decorate a
hint would hand a read-only command a `.objectstack/` side effect and a new way
to hang on an unreachable database, a poor trade for two words of status.
Status is served where a datastore is already open: both commands report before
they write anything, and the boot line carries the live verdict.

Building the advisory surfaced a defect in the gate that shipped with the scan.
`objectHasCoveredValueField` — the dormancy short-circuit meant to spare an
object with no covered field the flag query — tested raw type membership, while
the registry INJECTS covered-type fields into every object it registers:
`organization_id` and `owner_id` (both `system`), `created_by` and `updated_by`
(both in `SKIP_FIELDS`), four `lookup`s. `validateRecord` skips every one of
them before the value-shape check, so the short-circuit answered true for
literally every object, never fired, and its cache memoized a constant. It now
uses the validator's own `isScannableValueShapeField`, the predicate the
scanner already imports — three readings of "a covered field" drifting by one
clause is how a gate ends up governing fields nothing enforces. The flag read
was already memoized per process, so this corrects the property the ADR states
rather than a user-visible cost. The silent-when-uncovered test pins it: an
object declaring only text still carries those four injected lookups, and
counting by type made the advisory fire on every deployment that exists.

Verified: objectql 1326, cli 625, spec 7154, platform-objects 253,
service-storage 252, driver-memory 265, runtime 954; check:generated 8/8;
release-notes and doc-authoring gates pass; docs site and all 71 packages
build; ESLint clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zgA8CQMJeJbFEjnbib1Uv

Author: claude

.changeset/adr-0104-value-shape-scan-advertised.md

@@ -0,0 +1,42 @@
+---
+"@objectstack/objectql": patch
+"@objectstack/cli": patch
+---
+
+fix(migrate): the value-shape gate advertises itself, and stops counting fields nothing enforces (#3438)
+
+The scan landed with no way to find out it exists. ADR-0104's rollout asked for
+two advertisements and neither was built: `os migrate meta` — the command a
+16→17 upgrade already runs — never mentioned the data migrations, and a
+warn-first deployment booted silently. A gate nobody is told about is served by
+nobody, so the command shipped and the posture it governs never changed.
+
+`os migrate meta` now ends by naming both gated data migrations
+(`files-to-references`, `value-shapes`) with what each covers and the gate it
+records, on the human path and in `--json` alike, including the run that
+rewrote nothing — an upgrade needing no source change is exactly where "already
+done" is easy to assume. A kernel that boots with covered fields and no
+verified `adr-0104-value-shapes` row now logs one line naming the command that
+ends warn mode.
+
+Deliberately WITHOUT reading gate status from the datastore inside `os migrate
+meta`, which the ADR's sketch assumed. That command rewrites metadata SOURCE
+and has never opened a database — its e2e test runs the real CLI against a
+project in a tmpdir. Booting one to decorate a hint would hand a read-only
+command a `.objectstack/` side effect and a new way to hang on an unreachable
+database. Status is served where a datastore is already open: both commands
+report before they write anything, and the boot line carries the live verdict.
+
+The advisory also found a defect in the gate that shipped with the scan.
+`objectHasCoveredValueField` — the dormancy short-circuit that is supposed to
+spare an object with no covered field the flag query — tested raw type
+membership, while the registry INJECTS covered-type fields into every object it
+registers: `organization_id` and `owner_id` (both `system`), `created_by` and
+`updated_by` (both in `SKIP_FIELDS`), four `lookup`s. `validateRecord` skips
+every one before the value-shape check, so the short-circuit answered `true` for
+literally every object, never fired, and its cache memoized a constant. It now
+uses the validator's own `isScannableValueShapeField`, the same predicate the
+scanner imports — three readings of "a covered field" drifting by one clause is
+how a gate ends up governing fields nothing enforces. Behaviour is otherwise
+unchanged: the flag read was already memoized per process, so this corrects the
+property the ADR states rather than a user-visible cost.

content/docs/releases/v17.mdx

@@ -1012,6 +1012,14 @@ can correct — so it reports the object, field, type, count, sample record ids
 and parse issue, and you re-run until it is clean. `OS_ALLOW_LAX_VALUE_SHAPES=1`
 re-opens leniency while diagnosing.
 
+**You are told about both, rather than having to find this page.** `os migrate
+meta` — the command a 16→17 upgrade already runs — ends by naming the two data
+migrations and the gate each records, including on a run that rewrote nothing,
+since canonical metadata says nothing about stored values. A deployment that
+boots holding covered fields with no verified `adr-0104-value-shapes` row logs
+one line naming the command that ends warn mode. Neither reads as "done":
+until a gate is recorded, the classes it covers keep warning.
+
 **A database created by 17 attests both flags at creation** (#3438), so a new
 deployment enforces from its first boot instead of waiting for someone to run a
 migration that, for an empty store, does nothing. The platform attests only a

packages/cli/src/commands/migrate/meta.ts

@@ -24,6 +24,50 @@ import {
   emitJson,
 } from '../../utils/format.js';
 
+/**
+ * The gated ADR-0104 DATA migrations, named at the end of every `os migrate
+ * meta` run (#3438). This command is the one a 16→17 upgrade already runs, so
+ * it is where a deployment should learn that rewriting metadata is only half
+ * of the upgrade — a gate nobody is told about is served by nobody.
+ *
+ * Deliberately WITHOUT reading each gate's status from the datastore, which
+ * the ADR's sketch of this advisory assumed it would show. `os migrate meta`
+ * rewrites metadata SOURCE and has never opened a database — its e2e test
+ * spawns the real CLI against a project in a tmpdir. Booting one to decorate a
+ * hint would hand a read-only command a `.objectstack/` side effect and a new
+ * way to hang on an unreachable database, which is a poor trade for two words
+ * of status. Status is served where a datastore is already open: both commands
+ * report before they write anything, and a warn-first boot logs the
+ * value-shape line itself.
+ */
+const GATED_DATA_MIGRATIONS = [
+  {
+    command: 'os migrate files-to-references',
+    covers: 'file / image / avatar / video / audio values',
+    gate: 'adr-0104-file-references',
+  },
+  {
+    command: 'os migrate value-shapes',
+    covers: 'lookup / user / location / address / repeater / … value shapes',
+    gate: 'adr-0104-value-shapes',
+  },
+] as const;
+
+/** Print the data-migration advisory that closes a human-readable run. */
+function printDataMigrationAdvisory(): void {
+  console.log(chalk.bold('  Data migrations — metadata is only half of an upgrade:'));
+  for (const m of GATED_DATA_MIGRATIONS) {
+    console.log(`    • ${chalk.white(m.command)}`);
+    console.log(chalk.dim(`        covers: ${m.covers}`));
+    console.log(chalk.dim(`        gate:   ${m.gate}`));
+  }
+  console.log(
+    chalk.dim('    Each reports what it would do and writes nothing until `--apply`. Until a\n') +
+      chalk.dim('    deployment records a gate, the classes it covers stay warn-first (ADR-0104).'),
+  );
+  console.log('');
+}
+
 /**
  * `os migrate meta --from N` — replay the ADR-0087 D3 migration chain.
  *
@@ -112,6 +156,9 @@ export default class MigrateMeta extends Command {
                 : undefined,
               specChanges,
               schemaValid: parsed.success,
+              // The same advisory the human path prints, so an upgrading agent
+              // reading only JSON is told about the data half too (#3438).
+              dataMigrations: GATED_DATA_MIGRATIONS,
               duration: timer.elapsed(),
             });
         if (flags.out) writeFileSync(resolve(flags.out), JSON.stringify(result.stack, null, 2));
@@ -124,6 +171,11 @@ export default class MigrateMeta extends Command {
 
       if (result.applied.length === 0 && result.todos.length === 0) {
         printSuccess('Nothing to migrate — the metadata is already canonical for this range.');
+        // Still advertise: canonical METADATA says nothing about whether this
+        // deployment's stored VALUES have been scanned, and an upgrade that
+        // needed no source rewrite is exactly where that is easy to assume.
+        console.log('');
+        printDataMigrationAdvisory();
         return;
       }
 
@@ -171,6 +223,7 @@ export default class MigrateMeta extends Command {
         );
       }
       console.log('');
+      printDataMigrationAdvisory();
     } catch (error: any) {
       if (error instanceof MigrationFloorError) {
         if (flags.json) {

packages/cli/test/migrate-meta.e2e.test.ts

@@ -22,6 +22,7 @@ import { tmpdir } from 'node:os';
 import { join, resolve } from 'node:path';
 import { fileURLToPath } from 'node:url';
 import { ObjectStackDefinitionSchema } from '@objectstack/spec';
+import { PROTOCOL_MAJOR } from '@objectstack/spec/kernel';
 
 const execFileP = promisify(execFile);
 const HERE = resolve(fileURLToPath(import.meta.url), '..');
@@ -233,6 +234,41 @@ describe('os migrate meta --from 16 (e2e over the real CLI)', () => {
     }
   }, 120_000);
 
+  /**
+   * The upgrade-path advertisement (ADR-0104 rollout / #3438). Metadata is
+   * only half of a 16→17 upgrade, and this is the command that half runs
+   * through — so both surfaces name the data migrations, and both are pinned:
+   * an advertisement that silently stops appearing leaves the gate exactly as
+   * undiscoverable as having built no gate at all.
+   */
+  it('names both gated data migrations in the machine output', () => {
+    const commands = (out.parsed.dataMigrations ?? []).map((m: any) => m.command);
+    expect(commands).toContain('os migrate files-to-references');
+    expect(commands).toContain('os migrate value-shapes');
+    const gates = (out.parsed.dataMigrations ?? []).map((m: any) => m.gate);
+    expect(gates).toContain('adr-0104-value-shapes');
+  });
+
+  it('ends a human-readable run with the same advisory', async () => {
+    const stdout = await runMeta(['--from', '16'], dir);
+    expect(stdout).toContain('os migrate value-shapes');
+    expect(stdout).toContain('os migrate files-to-references');
+    // The advisory must not read as something the chain just did.
+    expect(stdout).toMatch(/only half of an upgrade/i);
+  }, 120_000);
+
+  it('advertises even when the chain had nothing to rewrite', async () => {
+    // The easiest run to mistake for "upgrade complete", and the one that
+    // returns before the normal tail: an empty chain (`--from` already at the
+    // runtime's major). Canonical METADATA says nothing about whether this
+    // deployment's stored VALUES have been scanned, so the advisory has to
+    // survive the early return too.
+    const stdout = await runMeta(['--from', String(PROTOCOL_MAJOR)], dir);
+    expect(stdout).toMatch(/Nothing to migrate/i);
+    expect(stdout).toContain('os migrate value-shapes');
+    expect(stdout).toContain('os migrate files-to-references');
+  }, 120_000);
+
   it('refuses a --from below the support floor with the structured error', async () => {
     let failed = false;
     try {

packages/objectql/src/engine.ts

@@ -12,7 +12,7 @@ import {
   type DroppedFieldsEvent
 } from '@objectstack/spec/data';
 import type { WriteObservabilityOptions } from '@objectstack/spec/contracts';
-import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, FILE_REFERENCE_TYPES, REFERENCE_VALUE_TYPES, STRUCTURED_JSON_TYPES, isFileIdToken, RAW_FILE_VALUES_CONTEXT_KEY } from '@objectstack/spec/data';
+import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, FILE_REFERENCE_TYPES, REFERENCE_VALUE_TYPES, isFileIdToken, RAW_FILE_VALUES_CONTEXT_KEY } from '@objectstack/spec/data';
 import {
   DATA_MIGRATION_FLAG_OBJECT,
   FILE_REFERENCES_MIGRATION_ID,
@@ -69,7 +69,7 @@ import { ExpressionEngine } from '@objectstack/formula';
 import type { Expression } from '@objectstack/spec';
 import { isAggregatedViewContainer, expandViewContainer } from '@objectstack/spec';
 import { bindHooksToEngine } from './hook-binder.js';
-import { validateRecord, normalizeMultiValueFields, coerceBooleanFields, ValidationError } from './validation/record-validator.js';
+import { validateRecord, normalizeMultiValueFields, coerceBooleanFields, ValidationError, valueShapePostureSetByEnv, isScannableValueShapeField } from './validation/record-validator.js';
 import { evaluateValidationRules, needsPriorRecord, stripReadonlyWhenFields, stripReadonlyWhenFieldsMulti, hasReadonlyWhenInPayload, stripReadonlyFields } from './validation/rule-validator.js';
 import { applyInMemoryAggregation } from './in-memory-aggregation.js';
 
@@ -2149,17 +2149,28 @@ export class ObjectQL implements IDataEngine {
   }
 
   /**
-   * Does this object declare a reference or structured-JSON field? Same
-   * per-schema cache and same dormancy rule as {@link objectHasMediaField}: an
-   * object holding none of these types can hold no violation of them, so it
-   * must not pay a query to learn that.
+   * Does this object declare a reference or structured-JSON field that a write
+   * would actually check? Same per-schema cache and same dormancy rule as
+   * {@link objectHasMediaField}: an object holding none can hold no violation
+   * of them, so it must not pay a query to learn that.
+   *
+   * The membership test is `isScannableValueShapeField` — the validator's own
+   * — and not raw type membership, because the registry INJECTS covered-type
+   * fields into every object it registers: `organization_id` and `owner_id`
+   * (both `system`), plus `created_by` / `updated_by` (both in `SKIP_FIELDS`),
+   * are all `lookup`s. `validateRecord` skips every one of them before it ever
+   * reaches the value-shape check, so counting them made this answer `true` for
+   * literally every object — the dormancy rule above never fired, and this
+   * cache memoized a constant. Same predicate as the scanner for the same
+   * reason the scanner imports it: three readings of "a covered field" drifting
+   * by one clause is how a gate ends up governing fields nothing enforces.
    */
   private objectHasCoveredValueField(objectSchema: any): boolean {
     if (!objectSchema?.fields) return false;
     const cached = ObjectQL.coveredValueFieldPresence.get(objectSchema);
     if (cached !== undefined) return cached;
-    const present = Object.values(objectSchema.fields).some(
-      (def: any) => def && (REFERENCE_VALUE_TYPES.has(def.type) || STRUCTURED_JSON_TYPES.has(def.type)),
+    const present = Object.entries(objectSchema.fields).some(
+      ([name, def]: [string, any]) => isScannableValueShapeField(name, def),
     );
     ObjectQL.coveredValueFieldPresence.set(objectSchema, present);
     return present;
@@ -2224,6 +2235,37 @@ export class ObjectQL implements IDataEngine {
     return this.valueShapesMigrationVerified;
   }
 
+  /**
+   * How many registered objects would benefit from being told about the
+   * value-shape scan at boot — or `null` when there is nothing worth saying.
+   *
+   * ADR-0104's rollout asks a warn-first deployment to log one startup line
+   * naming the command that ends warn mode, on the reasoning that a gate
+   * nobody is told about is served by nobody. Everything that would make that
+   * line false or useless answers `null` instead, cheapest test first so a
+   * kernel with nothing to advertise never reaches the flag query:
+   *
+   *  - an env switch already settled the posture (see
+   *    {@link valueShapePostureSetByEnv});
+   *  - no registered object declares a covered field, so the gate is moot here;
+   *  - the flag is verified, so the gate is already open.
+   *
+   * Read-only and advisory: it opens no gate and changes no posture. The count
+   * it returns is the reason the line is worth printing, so the operator can
+   * see the advice applies to their data rather than to the platform in
+   * general.
+   */
+  async valueShapeScanAdvisory(): Promise<number | null> {
+    if (valueShapePostureSetByEnv()) return null;
+    let objects: any[] = [];
+    try { objects = (this._registry as any).getAllObjects?.() ?? []; } catch { objects = []; }
+    let covered = 0;
+    for (const o of objects) if (this.objectHasCoveredValueField(o)) covered++;
+    if (covered === 0) return null;
+    if (await this.isValueShapesMigrationVerified()) return null;
+    return covered;
+  }
+
   /**
    * Read one deployment migration flag and answer whether it authorises its
    * consumers. Shared by both flags so the "every way of not knowing answers

packages/objectql/src/plugin.ts

@@ -543,6 +543,25 @@ export class ObjectQLPlugin implements Plugin {
         objectsRegistered: this.ql?.registry?.getAllObjects?.()?.length || 0
     });
 
+    // [ADR-0104 D1 / #3438] Tell a warn-first deployment that the gate exists.
+    // The engine decides whether there is anything to say (covered fields
+    // present, no env switch already settling the posture, flag unverified);
+    // every other case answers null and this stays silent. Wrapped because an
+    // advisory that fails a boot is worse than an advisory nobody reads — the
+    // flag read touches the database and a kernel mid-recovery must still start.
+    try {
+      const covered = await this.ql?.valueShapeScanAdvisory?.();
+      if (covered) {
+        ctx.logger.info(
+          `[value-shape] ${covered} registered object(s) declare reference or structured-JSON ` +
+            'fields and this deployment has no verified `adr-0104-value-shapes` row, so malformed ' +
+            'values WARN instead of being rejected. `os migrate value-shapes` reports what strict ' +
+            'mode would reject without writing anything; `--apply` records the gate once it is ' +
+            'clean (ADR-0104 / #3438).',
+        );
+      }
+    } catch { /* never fail a boot for an advisory */ }
+
     // ADR-0057: arm the periodic lifecycle sweep once the engine is live.
     this.lifecycleService?.start();
   }

packages/objectql/src/validation/record-validator.ts

@@ -594,6 +594,25 @@ function valueShapeStrictEffective(deploymentVerified: boolean): boolean {
   return deploymentVerified;
 }
 
+/**
+ * Has an environment switch already settled the reference / structured-JSON
+ * posture, so the boot advisory for the scan (#3438) would be untrue or
+ * useless?
+ *
+ * `OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1` means enforcement is already on for
+ * every class, so a line announcing "warn mode is in effect" would be false.
+ * `OS_ALLOW_LAX_VALUE_SHAPES=1` means the operator opted out deliberately —
+ * running the scan would not change what they get, so pointing at it is noise.
+ *
+ * Exported rather than re-read in the engine for the same reason the scanner
+ * imports `valueShapeViolation`: two readings of these variables drifting by
+ * one clause is how a deployment gets told to run a migration that would not
+ * change its posture, or gets told nothing while it still would.
+ */
+export function valueShapePostureSetByEnv(): boolean {
+  return LAX_VALUE_SHAPES() || VALUE_SHAPE_STRICT();
+}
+
 /**
  * Is `value` a violation of `def`'s declared stored shape — the SAME question
  * `validateOne` asks, exported so the `os migrate value-shapes` scanner counts