feat(migrate): os migrate value-shapes —— 引用/结构化 JSON 值形态的按部署门禁（#3438 第 2 项）

.changeset/adr-0104-value-shapes-scan-gate.md

@@ -0,0 +1,62 @@
+---
+"@objectstack/spec": minor
+"@objectstack/objectql": minor
+"@objectstack/cli": minor
+---
+
+feat(migrate): `os migrate value-shapes` — the per-deployment gate for reference and structured-JSON value shapes (#3438)
+
+The second of ADR-0104 D1's two evidence gates. Media value shapes already
+enforce once a deployment has verified its file migration (#3681); the
+reference (`lookup` / `master_detail` / `user` / `tree`) and structured-JSON
+(`location` / `address` / `composite` / `repeater` / `record` / `vector`)
+classes now get a gate of their own.
+
+```bash
+os migrate value-shapes           # scan: reports, writes nothing
+os migrate value-shapes --apply   # scan + record the deployment flag when clean
+```
+
+The run walks every stored value of those classes against
+`valueSchemaFor(field, 'stored')` — the same predicate the write path enforces,
+imported rather than re-derived — and, at zero violations, records
+`sys_migration { id: 'adr-0104-value-shapes', verified_at, blocking: 0 }`.
+Strict enforcement of these classes reads **that row**, never the platform
+version, so upgrading changes nothing until a deployment produces its own
+evidence.
+
+**There is no backfill, deliberately.** The file migration converts legacy
+values because the platform narrowed that storage form and owes the conversion.
+A malformed `location` is application data whose correct value only its author
+knows, so this run reports and prescribes — naming the object, field, type,
+count, offending record ids and the parse issue — and the operator fixes and
+re-runs. With nothing to convert, `--apply`'s only write is the flag row, which
+keeps the #3617 invariant trivially: a dry run changes nothing, and whether a
+run changed this deployment's posture never depends on what it found.
+
+**A separate flag from the file migration**, because it attests a separate
+fact. That flag says file values were migrated and their ownership reconciled;
+it says nothing about whether a `lookup` id or a `location` payload is well
+formed. Gating these classes on it would be borrowing evidence for a fact it
+does not cover.
+
+- New escape hatch **`OS_ALLOW_LAX_VALUE_SHAPES=1`** returns a verified
+  deployment to warnings, with the same precedence as its media sibling: the
+  opt-out beats `OS_DATA_VALUE_SHAPE_STRICT_ENABLED`, which beats the flag.
+  Wrongly staying lenient costs a warning; wrongly enforcing stops a working
+  app from writing.
+- `@objectstack/spec/system` exports `VALUE_SHAPES_MIGRATION_ID`.
+- `@objectstack/objectql` exports `scanValueShapes`, `valueShapeScanPassed`
+  and `formatValueShapeScanReport`. The scanner is read-only and does **not**
+  record the flag: readers of a migration flag use the spec contract, only
+  writers depend on `@objectstack/platform-objects`, so the composition lives
+  with the CLI command rather than inverting the engine's dependencies.
+- `validateRecord` gains `valueShapeStrict`, the sibling of
+  `mediaValueShapeStrict`. Both default to `false`: a caller that cannot say
+  stays lenient, so nothing starts rejecting merely because the evidence was
+  unavailable.
+
+**Nothing changes for an existing deployment until it runs the command.** A
+scan that is truncated, or that cannot read an object, fails the gate even with
+zero violations found — "none in the part we read" is not the claim the flag
+makes.

content/docs/deployment/cli.mdx

@@ -628,6 +628,7 @@ where the data lives.
 | Command | Description |
 |---------|-------------|
 | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag |
+| `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean |
 
 ```bash
 os migrate files-to-references            # Dry run: full report, writes nothing
@@ -660,8 +661,9 @@ Exit status is `0` only when the self-check passes, so CI can gate on it.
 | **Released-file collection** | A field file whose one owning record lets go (the field is cleared or the record deleted) is tombstoned into the declared 30-day grace window; re-referencing the id within the window revives it, and after it the platform sweep reclaims the row and its bytes. Unverified deployments keep every released file forever. |
 
 Other value classes are unaffected: a `lookup` or `location` value keeps its own
-warn-first rollout, because this migration is evidence about *file* values and
-says nothing about theirs.
+warn-first rollout until `os migrate value-shapes` (below) supplies *their*
+evidence, because this migration is evidence about *file* values and says
+nothing about theirs.
 
 <Callout type="warn">
 A dry run writes **nothing** — not the conversions, and not the flag either,
@@ -675,6 +677,53 @@ delete check re-reads the flag fresh, so a later failing run stops collection
 without a restart.
 </Callout>
 
+#### `os migrate value-shapes`
+
+The same gate for the **non-media** value classes — references (`lookup`,
+`master_detail`, `user`, `tree`) and structured JSON (`location`, `address`,
+`composite`, `repeater`, `record`, `vector`).
+
+```bash
+os migrate value-shapes                    # Scan: full report, writes nothing
+os migrate value-shapes --apply            # Scan, then record the flag if clean (prompts)
+os migrate value-shapes --apply --yes --json   # CI / scripts
+os migrate value-shapes --object contact       # Restrict to one object (repeatable)
+```
+
+**This one converts nothing.** Its sibling rewrites legacy file values because
+the platform narrowed that storage form and therefore owes the conversion; a
+`location` stored as `{latitude, longitude}` instead of `{lat, lng}` is
+*application* data whose correct value only its author knows. So the run reports
+— object, field, type, how many records, sample record ids, and the parse issue
+— and you fix the values (or the code writing them) and re-run until it is
+green. Because there is nothing to convert, `--apply`'s only write is the flag
+row itself.
+
+A scan that is **truncated** (by `--max-records`) or that cannot read an object
+fails the gate even with zero violations found: "none in the part we read" is
+not the claim the flag makes.
+
+| Once verified | Effect |
+| :--- | :--- |
+| **Reference + structured-JSON value shapes** | A malformed value of those classes is **rejected** (`400 invalid_type`) instead of warned about. Set `OS_ALLOW_LAX_VALUE_SHAPES=1` to re-open leniency while diagnosing. |
+
+This flag is deliberately **separate** from the file migration's. That one
+attests that file values were migrated and their ownership reconciled — it says
+nothing about whether a `lookup` id or a `location` payload is well formed, so
+it may not vouch for these classes. A deployment can legitimately have passed
+either without the other.
+
+<Callout type="warn">
+`OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1` turns on **every** value class at once,
+regardless of which migrations this deployment has run. It is the "I already
+know my data" lever, not the route to strictness — the route is running the
+migration that produces the evidence.
+</Callout>
+
+Same writing rules as its sibling: a dry run writes **nothing**, `--apply` is
+the only writing mode, a later failing run clears the verified state, and a
+running server reads the flag once — **restart it** after a successful apply.
+
 #### A database created by this version needs no migration
 
 A deployment whose database the platform **creates from empty** records these
@@ -691,8 +740,10 @@ attests nothing and produces its evidence by running the command, because
 
 Importing legacy values into such a deployment is rejected at the write path
 rather than silently accepted. That is the intended outcome; if you must admit
-them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` re-opens leniency, and
-re-running the migration re-establishes the flag from the data itself.
+them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` (media) and
+`OS_ALLOW_LAX_VALUE_SHAPES=1` (references and structured JSON) re-open leniency
+per class, and re-running the corresponding migration re-establishes its flag
+from the data itself.
 
 ### Scaffolding
 

content/docs/releases/v17.mdx

@@ -984,6 +984,23 @@ records** — never the platform version — is what authorises both strict medi
 value shapes and irreversible file collection. Upgrading changes neither;
 running the migration does.
 
+**The non-media classes have their own gate** (#3438) — references (`lookup`,
+`master_detail`, `user`, `tree`) and structured JSON (`location`, `address`,
+`composite`, `repeater`, `record`, `vector`):
+
+```bash
+os migrate value-shapes           # scan: reports, writes nothing
+os migrate value-shapes --apply   # scan, then record the flag if clean
+```
+
+A separate flag because it attests a separate fact: the file migration says file
+values were converted and their ownership reconciled, which tells you nothing
+about whether a `lookup` id or a `location` payload is well formed. This one
+converts nothing — a malformed `location` is application data only its author
+can correct — so it reports the object, field, type, count, sample record ids
+and parse issue, and you re-run until it is clean. `OS_ALLOW_LAX_VALUE_SHAPES=1`
+re-opens leniency while diagnosing.
+
 **A database created by 17 attests both flags at creation** (#3438), so a new
 deployment enforces from its first boot instead of waiting for someone to run a
 migration that, for an empty store, does nothing. The platform attests only a
@@ -1306,7 +1323,11 @@ objectui commits on top of the pin 16.1.0 shipped.
   media value shapes *and* released-file collection for this deployment, so
   read the report before you `--apply`. Do **not** reach for
   `OS_DATA_VALUE_SHAPE_STRICT_ENABLED` to get there: it opts every value class
-  in at once, including ones with no migration behind them.
+  in at once, regardless of which migrations this deployment has actually run.
+- **Reference and structured-JSON values:** run `os migrate value-shapes` and
+  fix what it reports before `--apply`. It converts nothing — the values it
+  names are application data — and a scan that was truncated or could not read
+  an object fails the gate even at zero violations.
 - **Datasources:** verify every declared datasource connects in every
   environment — a bound datasource that cannot connect now fails the boot
   instead of failing every later query.

packages/cli/src/commands/migrate/files-to-references.ts

@@ -17,7 +17,7 @@ import {
 import { bootSchemaStack } from '../../utils/schema-migrate.js';
 import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js';
 import { describeOccupancy } from '../../utils/sqlite-occupancy.js';
-import { resolveStorageCapabilityArg } from '../serve.js';
+import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js';
 
 async function confirm(question: string): Promise<boolean> {
   if (!process.stdin.isTTY) return false; // non-interactive → require --yes
@@ -30,36 +30,6 @@ async function confirm(question: string): Promise<boolean> {
   }
 }
 
-/**
- * The settings + storage service plugins, so the booted kernel carries
- * `sys_file`/`sys_migration` and the deployment's REAL storage adapter.
- * Settings first: the storage plugin re-resolves its adapter from persisted
- * settings when a settings service is present, which is how an S3-configured
- * deployment's backfill uploads land in S3 rather than on this machine.
- * Storage config goes through the SAME resolver `os serve` uses
- * (`resolveStorageCapabilityArg`), so the CLI materialises bytes exactly where
- * the server would. It did not, and that mattered here more than anywhere: this
- * command reconciles what records claim against what storage actually holds, so
- * a root that disagrees with the server's reconciles against the wrong tree.
- * It built `{ driver: 'local', root }` — keys `StorageServicePluginOptions` does
- * not declare — so the adapter silently used the plugin's own `./storage`
- * default while the server (since framework#4096) writes under
- * `.objectstack/data/uploads`.
- */
-async function buildDataMigrationPlugins(): Promise<unknown[]> {
-  const plugins: unknown[] = [];
-  try {
-    const { SettingsServicePlugin } = await import('@objectstack/service-settings');
-    plugins.push(new SettingsServicePlugin({ registerRoutes: false }));
-  } catch {
-    // optional — without it, constructor/env-driven storage config still applies
-  }
-  const { StorageServicePlugin } = await import('@objectstack/service-storage');
-  const { options } = resolveStorageCapabilityArg(process.env.OS_STORAGE_ROOT);
-  plugins.push(new StorageServicePlugin({ ...options, registerRoutes: false }));
-  return plugins;
-}
-
 /**
  * `os migrate files-to-references` — the ADR-0104 D3 data migration, with its
  * self-check gate (#3617).