feat(migrate): gate the file-as-reference migration per deployment, not per release (#3617)

The ADR-0104 D3 wave 2 acceptance gate was written as an operations runbook —
run the backfill on the tenant, watch `verifyFileReferences` stay clean for
seven days, then flip the switch. Every clause assumes we run the tenant, we
observe it, and we flip.

ObjectStack is a development platform. Third-party deployments upgrade on their
own schedule and their data is not visible to us, so there is no observation
window of ours that anyone else's data can wait inside. A gate phrased as one is
not strict, merely unlocatable.

R4's substance is unchanged — a ledger may not be given the power to delete
bytes until it has been shown to agree with reality. What changes is who
produces the evidence: each deployment, for itself.

  os migrate files-to-references            # dry run: reports, writes nothing
  os migrate files-to-references --apply    # converts, verifies, records the flag

An apply run whose reconciliation reports zero blocking discrepancies records
`sys_migration { id: 'adr-0104-file-references', verified_at, blocking: 0 }`.
That row — never the platform version — is what may later open released-file
collection (#3459 PR-5b) and strict media value-shape enforcement (#3438); both
read the same flag through one spec-level predicate so the two gates cannot
disagree about the same fact. Not run, or not passed, leaves files retained
forever: storage cost, zero data loss. A later failing run clears the verified
state, so a database that has drifted closes its own gate.

A dry run writes nothing at all — not the conversions, and not the flag either,
even when the self-check would pass — so whether a run changed the deployment's
posture never depends on what it happened to find. The command also refuses to
run when no app metadata is loaded: an empty scan is indistinguishable from a
clean one, and this verdict authorises irreversible behaviour.

Also fixes a defect the end-to-end run surfaced: on a live kernel the engine's
read resolver expanded stored ids to their `{ id, url, … }` form before the
scan saw them, so the reconciliation reported every held reference as absent —
noisy `stale_owner` findings, and a missed `unowned_reference` would have been a
false pass of the very gate this work builds. Reads may now opt out via the
spec's `RAW_FILE_VALUES_CONTEXT_KEY`; the storage service's scan and bookkeeping
reads do.

Verified end-to-end against the showcase app on SQLite: 10 legacy values
converted and claimed, reconciliation clean, flag recorded; a dry run against
the same database left both the data and the flag untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V9uWWyKq6pNPQthwCXL8ma

Author: claude

.changeset/adr-0104-deployment-migration-gate.md

@@ -0,0 +1,58 @@
+---
+"@objectstack/spec": minor
+"@objectstack/platform-objects": minor
+"@objectstack/service-storage": minor
+"@objectstack/objectql": minor
+"@objectstack/cli": minor
+---
+
+feat(migrate): `os migrate files-to-references` — a data migration with a self-check, gated per deployment (#3617)
+
+The ADR-0104 file-as-reference migration ships as a command a deployment runs
+against its own database, and the deployment-level flag it records is what may
+later authorise irreversible behaviour — never the platform version.
+
+```bash
+os migrate files-to-references           # dry run: reports, writes nothing
+os migrate files-to-references --apply   # converts, verifies, records the flag
+```
+
+The run backfills legacy file-field values (inline metadata blobs, own-resolver
+URLs, `data:` URIs) into owned `sys_file` references, reconciles the ownership
+ledger against what records actually hold, and — only on an `--apply` run whose
+reconciliation reports **zero blocking discrepancies** — records
+`sys_migration { id: 'adr-0104-file-references', verified_at, blocking: 0 }`.
+
+**Why a flag rather than a release note.** ObjectStack is a development
+platform: third-party deployments upgrade on their own schedule and their data
+is not observable by anyone else, so no release-side soak can vouch for them.
+The evidence has to be produced where the data is. Consequences:
+
+- Installing a new version never starts deleting bytes. Running the migration
+  and passing its self-check is the consent.
+- Not run, or not passed → files are retained forever. Wasted storage, zero
+  data loss.
+- A later failing run **clears** `verified_at`: a deployment whose data has
+  drifted closes its own gate.
+- A dry run writes nothing at all — not the conversions, and not the flag,
+  even when the self-check would pass.
+- External URLs stay advisory. They are not `sys_file`s, so they can never
+  enter collection; whether to remodel them as a `url` field is the app
+  author's decision (ADR-0104 R7), not a gate.
+
+Ships alongside:
+
+- `@objectstack/spec` — `DataMigrationFlagSchema`, `FILE_REFERENCES_MIGRATION_ID`,
+  and the single `isDataMigrationFlagVerified` predicate both future consumers
+  (collection #3459, strict value-shape #3438) read, so the two gates cannot
+  disagree about the same fact.
+- `@objectstack/platform-objects` — the `sys_migration` object plus
+  `readDataMigrationFlag` / `isDataMigrationVerified` / `recordDataMigrationRun`.
+  Reads fail toward "not verified": a gate that cannot read its evidence stays
+  closed.
+- `@objectstack/objectql` — a read may now opt out of file-reference expansion
+  via the spec's `RAW_FILE_VALUES_CONTEXT_KEY`, and the storage service's
+  bookkeeping/scan reads do. Without it the read resolver rewrites stored ids to
+  their expanded form before the reconciliation sees them, which reports held
+  references as absent — noisy `stale_owner` findings, and a missed
+  `unowned_reference` would have been a false pass of the collection gate.

content/docs/deployment/cli.mdx

@@ -517,6 +517,48 @@ first. It never drops a table that is absent from your metadata, and on SQLite
 it reconciles via a table rebuild (copy → swap) that preserves your data.
 </Callout>
 
+#### Data migrations
+
+The commands above reconcile **schema**. A *data* migration rewrites rows, and
+whether it is done is a fact about **your** database, not about the platform
+version you installed — so each deployment runs it, and its result is recorded
+where the data lives.
+
+| Command | Description |
+|---------|-------------|
+| `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag |
+
+```bash
+os migrate files-to-references            # Dry run: full report, writes nothing
+os migrate files-to-references --apply    # Convert, verify, record the flag (prompts)
+os migrate files-to-references --apply --yes --json   # CI / scripts
+os migrate files-to-references --object product       # Restrict to one object (repeatable)
+```
+
+A `file` / `image` / `avatar` / `video` / `audio` field value is an opaque
+`sys_file` id that the platform owns. Values written before that (an inline
+`{url, name, …}` blob, or a URL naming this platform's own
+`…/storage/files/:id` resolver) are converted in place; **external URLs are
+reported, never re-hosted** — re-hosting third-party content is a licensing and
+privacy decision, and the right fix is usually to model the field as a `url`
+field instead.
+
+The run then reconciles what records actually hold against what `sys_file`
+records as each file's owner. Zero blocking discrepancies is what records the
+flag — and that flag, not the version number, is what later enables features
+that act irreversibly on migrated data (reclaiming the bytes of released files;
+rejecting rather than warning about legacy media values). Never running it is
+safe: files are simply retained forever.
+
+Exit status is `0` only when the self-check passes, so CI can gate on it.
+
+<Callout type="warn">
+A dry run writes **nothing** — not the conversions, and not the flag either,
+even when the self-check would pass. `--apply` is the only writing mode. A
+later run that *fails* its self-check clears the flag's verified state, so a
+database that has drifted closes its own gate.
+</Callout>
+
 ### Scaffolding
 
 | Command | Alias | Description |

docs/adr/0104-field-runtime-value-shape-contract.md

@@ -591,10 +591,13 @@ legacy values would reject any update that rewrites such a field, breaking
 working apps until the backfill catches up. Backfilling first leaves the cutover
 nothing legacy to reject.
 
-R4's acceptance gate is now executable rather than aspirational: step 6 may not
-merge until step 5 reports **zero blocking discrepancies for ≥7 consecutive
-days** on real tenant data. R5 (public-posture inventory) and R6 (sub-key read
-scan) stand.
+R4's acceptance gate is now executable rather than aspirational: step 5's
+reconciliation must report **zero blocking discrepancies** before step 6 may
+act. (The original wording — "for ≥7 consecutive days on real tenant data" —
+assumed a tenant we operate and observe; see the 2026-07-27 platform-form
+addendum below, which relocates the same evidence requirement to the one place
+that can satisfy it.) R5 (public-posture inventory) and R6 (sub-key read scan)
+stand.
 
 Steps 4 and 6 are the two that can break something. Step 4 is a declared
 protocol major and breaks *loudly* — a rejected write, recoverable. Step 6
@@ -604,15 +607,86 @@ adoption that consumes the new form, while step 6 additionally requires the
 reconciliation evidence above, and neither the enable nor the migration run
 should be performed without an explicit human decision.
 
+## Addendum (2026-07-27, later) — the evidence gate is per deployment, not per release
+
+The gate above was written as an operations runbook: *run the backfill on the
+tenant, watch `verifyFileReferences` stay clean for a week, then flip the
+switch.* Every clause of that sentence assumes **we** run the tenant, **we**
+observe it, and **we** flip.
+
+ObjectStack is a development platform. Third-party developers build metadata
+apps on it and ship versions onward; each deployment decides for itself when to
+upgrade, and its data is not visible to us — nor should it be. **There is no
+observation window of ours that anyone else's data can wait inside.** A gate
+phrased as one is not strict, merely unlocatable: it can be satisfied by no
+deployment at all, including the deployments whose bytes are at stake.
+
+### What survives, and what moves
+
+R4's substance is unchanged: **a ledger may not be given the power to delete
+bytes until it has been shown to agree with reality.** What changes is *who
+produces the evidence* — from us, once, to each deployment, for itself.
+
+That rules out a one-shot script (a script whose output nobody must read is not
+a gate) and rules in a **data-migration step with a self-check, whose passing
+result is recorded on the deployment**:
+
+```
+os migrate files-to-references
+  1. backfill                 (dry run by default; --apply writes)
+  2. verifyFileReferences     (reconcile the ledger against what records hold)
+  3. zero blocking findings →  record sys_migration { id: 'adr-0104-file-references',
+                                 verified_at, blocking: 0 }
+  4. collection + strict enforcement read THAT ROW, never the version number
+```
+
+### The properties this buys
+
+- **Upgrading does not start deleting.** Installing a new version is not
+  consent; running the migration and passing its self-check is.
+- **A third-party developer need not understand R4.** They run one command.
+  Green means done; red names the record holding a file nobody owns.
+- **Not run, or not passed → files live forever.** Wasted storage, zero data
+  loss: "fail toward retention" holds on the *deployment* axis too. The same
+  rule applies to a regression — a later failing run clears `verified_at`, so a
+  deployment whose data has drifted closes its own gate.
+- **Each deployment gets its own 30-day tombstone window.** Nobody waits inside
+  anyone else's soak. The irreversible moment is day 30 after *that* deployment
+  enabled collection, not release day.
+- **A dry run changes nothing** — not the data, and not the flag either, even
+  when the self-check would pass. `--apply` is the only writing mode, so
+  "did this run change my posture" never depends on what the run found.
+
+### D1/D2 carry the identical error, and the identical fix
+
+The strict-by-default flip for value-shape and action-param validation (#3438)
+was scheduled as "flip in a later minor once telemetry is quiet" — again
+*our* telemetry, deciding for *their* deployments. A deployment that has not
+finished backfilling would have every media-field update rejected the moment it
+upgraded: a working app broken by a decision made on its behalf.
+
+Both therefore read the **same** flag. Strict enforcement becomes effective for
+a deployment when that deployment has completed and verified its migration —
+not when a version number arrives. One flag, not two gates that can disagree:
+they are gating on the same fact.
+
+### What cannot be automated, and does not block
+
+Whether an **external URL** (`https://cdn…`) should become an explicit `url`
+field is a modelling decision only the app's author can make (R7). It also
+cannot block: an external URL is not a `sys_file` and can never enter
+collection. It is reported as advisory, never as a gate failure.
+
 ### Why this stays inside ADR-0104 rather than a new ADR
 
 D3 is already this ADR's third phase; the two-wave split and the
 enforcement-point principle are a **refinement of the D4 rollout**, not a new
 decision, so they live here. Wave 2's migration mechanics (dual-read window,
 `os migrate` backfill, the R4/R5/R6 gates) are specified in §D3 above and need
-no separate record. Wave 2's implementation did surface one genuinely new
-decision — the exclusive-ownership model — and it is recorded as the 2026-07-27
-addendum above rather than a separate ADR, because it settles *how* D3's
-already-accepted "field values point into `sys_file`" behaves rather than
-revisiting whether to do it. A future choice that stands on its own (say, a
-chunked-migration protocol, or byte-layer content dedup) would earn its own ADR.
+no separate record. Wave 2's implementation did surface two genuinely new
+decisions — the exclusive-ownership model, and relocating the evidence gate to
+the deployment — and both are recorded as 2026-07-27 addenda above rather than
+separate ADRs, because each settles *how* D3's already-accepted "field values
+point into `sys_file`" reaches production rather than revisiting whether to do
+it. A future choice that stands on its own (say, a chunked-migration protocol,
+or byte-layer content dedup) would earn its own ADR.