feat(migrate,metadata-protocol): os migrate meta --stored rewrites sys_metadata rows in place (#4327) (#4464)

* feat(migrate,metadata-protocol): os migrate meta --stored rewrites sys_metadata rows in place (#4327)

#4317 closed the correctness gap from the read side: every stored-row
rehydration seam replays the full ADR-0087 conversion chain, retired entries
included, so a row written under any past protocol is served canonical
forever. The rows themselves stayed legacy — the chain re-lowers them on every
load and each logs a conversion notice per process. Until now the only things
that rewrote such a row were a Studio re-save and duplicatePackage.

`os migrate meta --stored` walks sys_metadata (active + draft, all orgs),
replays the same applyConversionsToStoredItem pass, and re-saves each changed
body through saveMetaItem — so a rewritten row gets a sys_metadata_history
entry, a fresh checksum and the mutation projectors, exactly like an author's
save. The history row's source is `migrate-stored`, distinguishing an upgrade
from an edit. parentVersion is the row's own checksum, so a concurrent writer
produces a 409 the report names rather than a clobber.

Preview is the default and --apply the only writing mode, matching its two
siblings and #3617's "a dry run changes nothing"; an apply run refuses to
start while another process holds the SQLite database. Nothing gates on this
having run (#3855) and no sys_migration flag is recorded — a flag would
advertise enforcement that does not exist. What a run buys is hygiene plus an
assertable verdict: nothing left to do exits 0, work remaining exits 1.

Three carve-outs are reported rather than counted as done: flow rows (their
seam is AutomationEngine.registerFlow, which holds the executor registry the
node-type conflict guard needs), types with no repository write path (agent),
and rows that still fail the current schema after conversion. An empty scan
says it attests nothing rather than reading as a pass.

Also: protocol.migrateStoredMetadata() returns the same structured report an
admin route would render, and saveMetaItem takes an optional `source` for its
history/audit rows — server-stated, never request-derived.

Closes #4327

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WoZPKPDqJ7WB7z84xk9y3f

* docs(adr-0087): name the follow-up that gives flow rows the same finish line (#4454)

The addendum said flows were "tracked separately" without saying where.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WoZPKPDqJ7WB7z84xk9y3f

* docs(releases): name the stored-metadata pass on the v17 page (#4327)

The v17 entry for #3903 described the read-path guarantee and stopped there,
and the upgrade checklist listed the two per-deployment migrations without
this one. Both now point at `os migrate meta --stored`, marked optional —
it opens no gate, unlike its two neighbours in that list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WoZPKPDqJ7WB7z84xk9y3f

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/migrate-meta-stored-rewrite.md

@@ -0,0 +1,62 @@
+---
+"@objectstack/metadata-protocol": minor
+"@objectstack/cli": patch
+---
+
+feat(migrate,metadata-protocol): `os migrate meta --stored` rewrites sys_metadata rows so the read-path chain has a finish line (#4327)
+
+#4317 closed the correctness gap from the read side: every stored-row
+rehydration seam replays the full ADR-0087 conversion chain, retired entries
+included, so a row written under any past protocol is *served* canonical
+forever. What it deliberately did not do is make the rows themselves canonical.
+A pre-17 row keeps its legacy bytes, the chain re-lowers it on every load, and
+each affected row logs one conversion notice per process — deduped, but back
+every boot. Until now the only things that ever rewrote such a row were a Studio
+re-save and `duplicatePackage`.
+
+**`os migrate meta --stored`** is the pass that ends it for a deployment that
+runs it. It walks `sys_metadata` — `active` and `draft`, every organization —
+replays the same `applyConversionsToStoredItem` chain, and re-saves each changed
+body through the normal write path, so a rewritten row gets a
+`sys_metadata_history` entry, a fresh checksum and the mutation projectors,
+exactly like an author's save. The history row's `source` is `migrate-stored`,
+so a later diff distinguishes an upgrade from somebody's edit.
+
+```bash
+os migrate meta --stored                    # preview: per-row report, writes nothing
+os migrate meta --stored --apply            # rewrite the rows (prompts)
+os migrate meta --stored --apply --yes --json   # CI / scripts
+os migrate meta --stored --type view        # restrict to a type (repeatable)
+```
+
+**Preview is the default and `--apply` is the only writing mode** — the house
+rule its siblings already keep (#3617's "a dry run changes nothing"), and it
+applies with more force here because what moves is metadata: every affected
+row's checksum and a history entry per row. An apply run also refuses to start
+while another process holds the SQLite database, for the same reason
+`os migrate files-to-references --apply` does.
+
+**Nothing gates on this having run.** #3855's conclusion stands — an
+operator-run migration cannot be relied upon, so the read path remains the
+guarantee for every deployment, and no `sys_migration` flag is recorded (a flag
+would advertise enforcement that does not exist). What a run buys is hygiene —
+rows stop carrying pre-protocol dialects, so diffs, exports and history are
+clean going forward, and the recurring notices go quiet — plus one thing that
+was previously unobtainable: **an operator can assert it.** A run with nothing
+left to do exits `0`, a deployment with rows still on an old dialect exits `1`,
+so "my metadata is on protocol N" becomes a CI check rather than a belief.
+
+Three things the pass declines, and reports rather than counting as done:
+`flow` rows (their seam is `AutomationEngine.registerFlow`, which holds the
+executor registry the node-type conflict guard needs), types with no repository
+write path (`agent` — rewriting there would record no history and force a draft
+live), and rows that still fail the current schema after conversion (a genuine
+contract violation the write path is right to refuse; it keeps reading through
+the chain and stays fixable in Studio).
+
+Also new, and usable without the CLI: `protocol.migrateStoredMetadata()` returns
+the same structured report an admin route would render, and `saveMetaItem`
+accepts an optional `source` for the history/audit rows. `source` is not
+request-derived — the REST layer builds its save request field by field and
+never forwards a client-supplied value, so provenance stays something the server
+states rather than something a caller claims.

content/docs/deployment/cli.mdx

@@ -589,6 +589,7 @@ written.
 | `os migrate plan` | Warns and continues — a plan writes nothing either way |
 | `os migrate apply` | **Refuses** (exit 1, `error: database_busy` under `--json`). Stop the other process, or pass `--force` |
 | `os migrate files-to-references --apply` | **Refuses** likewise — it rewrites rows, so a concurrent writer is at least as dangerous |
+| `os migrate meta --stored --apply` | **Refuses** likewise — it rewrites `sys_metadata` rows, and a live process saving metadata is exactly the collision |
 
 The check applies to SQLite only: Postgres and MySQL take their own server-side
 locks. Only same-user processes are visible without elevated privileges, and a
@@ -642,6 +643,7 @@ where the data lives.
 |---------|-------------|
 | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag |
 | `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean |
+| `os migrate meta --stored` | Replay the metadata conversion chain over this deployment's `sys_metadata` rows and rewrite the ones still carrying a pre-protocol shape. Hygiene, not a gate — nothing depends on it having run |
 
 ```bash
 os migrate files-to-references            # Dry run: full report, writes nothing
@@ -774,6 +776,64 @@ closed gate logs that it is enforcing, and an app that declares neither class
 of field says nothing at all. So a running deployment always tells you the
 state of its own data — which is the question `os migrate meta` cannot answer.
 
+#### `os migrate meta --stored`
+
+The two commands above are about **application data**. This one is about the
+**metadata itself**, at rest: the `sys_metadata` rows Studio and the runtime
+authoring APIs write.
+
+Those rows already *read* correctly whatever protocol they were written under —
+every rehydration seam replays the full conversion chain, so a body from an
+older major is served in today's canonical shape and always will be. What the
+rows do not do is *change*: they keep their original bytes, the chain re-lowers
+them on every load, and each one logs a conversion notice once per boot. This
+command ends that for the deployment that runs it.
+
+```bash
+os migrate meta --stored                    # Preview: per-row report, writes nothing
+os migrate meta --stored --apply            # Rewrite the rows (prompts)
+os migrate meta --stored --apply --yes --json   # CI / scripts
+os migrate meta --stored --type view --type object   # Restrict to a type (repeatable)
+```
+
+It walks `active` and `draft` rows across every organization (archived rows are
+a record of what *was* and are never read), replays the same chain the read path
+does, and re-saves each changed body through the normal write path — so a
+rewritten row gets a `sys_metadata_history` entry, a fresh checksum, and the
+mutation projectors, exactly like an author's save. The history entry's source
+is `migrate-stored`, so a later diff shows which changes were an upgrade and
+which were somebody's edit.
+
+Three things it deliberately declines, and names in the report rather than
+counting as done:
+
+| Not rewritten | Why |
+| :--- | :--- |
+| `flow` rows | Flow-node conversions carry a conflict guard that needs the automation engine's live executor registry; flows canonicalize at their own seam when the engine loads them |
+| Types with no repository write path (`agent`) | Their write path records no history and would force a draft live — a half-write is worse than leaving the row to the read path |
+| Rows that still fail the current schema after conversion | That is a genuine contract violation, not chain-owned history. The write path's rejection is correct; fix the row in Studio |
+
+<Callout type="warn">
+`--apply` is the only writing mode, and it rewrites **metadata** — each affected
+row's checksum moves and each gets a history entry. Preview first. Like the
+other row-rewriting migration, an apply run refuses to start while another
+process holds the SQLite database (`--force` overrides).
+</Callout>
+
+**Nothing gates on this having run.** The read path is the guarantee, for every
+deployment, whether or not anyone runs this — an operator-run migration is not
+something the platform can depend on. What running it buys is hygiene (cleaner
+diffs, exports and history from here on, and the recurring boot notices go
+quiet) plus one thing that was previously unobtainable: **you can assert it.**
+A run with nothing left to do exits `0`; a deployment with rows still carrying
+an old dialect exits `1`. So "my metadata is on protocol N" becomes a check
+rather than a belief.
+
+Note the division of labour with the default mode: `os migrate meta --from N`
+rewrites an **author's source** and reads no database; `--stored` rewrites **one
+deployment's rows** and reads no config. Same chain, opposite ends of the
+contract — which is why the two modes are mutually exclusive.
+
 ### Scaffolding
 
 | Command | Alias | Description |

content/docs/releases/v17.mdx

@@ -1716,7 +1716,12 @@ platform capabilities an administrator gains, then the Console delta.
   source at every read seam (including `registerFlow` rehydration) — a stored
   action with the removed `execute` dispatches via `target` again; boot
   hydration validates each row post-conversion and diagnoses invalid ones
-  with `[metadata_spec_invalid]` instead of shrugging.
+  with `[metadata_spec_invalid]` instead of shrugging. **The rows themselves
+  can now be brought forward too (#4327):** `os migrate meta --stored`
+  replays the chain over `sys_metadata` and rewrites what still carries a
+  pre-protocol shape, through the normal write path. Optional — the read path
+  is the guarantee either way — but it is what makes the conversion pass a
+  no-op on your data instead of a permanent shim.
 - **Studio's metadata forms tell the truth (#3786).** Four of seventeen forms
   had drifted from their schemas, so controls saved nothing: the Object →
   Capabilities toggles bound a key the schema does not declare (all seven
@@ -2021,6 +2026,12 @@ covers are folded into the list below rather than left to the changelog.)
   fix what it reports before `--apply`. It converts nothing — the values it
   names are application data — and a scan that was truncated or could not read
   an object fails the gate even at zero violations.
+- **Stored metadata (optional):** run `os migrate meta --stored` to see which
+  `sys_metadata` rows still carry a pre-17 shape, and `--apply` to rewrite
+  them. Unlike the two above this opens no gate and nothing depends on it —
+  those rows already read canonically, forever. It stops them re-converting on
+  every load, keeps diffs and exports clean going forward, and gives you an
+  exit code to assert on: nothing left to do exits `0`.
 - **Datasources:** verify every declared datasource connects in every
   environment — a bound datasource that cannot connect now fails the boot
   instead of failing every later query.

docs/adr/0087-metadata-protocol-upgrade-contract.md

@@ -421,3 +421,39 @@ diverged (#3903). This addendum extends the contract to data at rest:
   at boot would unhook live tables and make the row unfixable in Studio
   (availability over purity for data at rest; the same verdict reaches Studio
   as `_diagnostics` on every read).
+
+## Addendum (2026-08-01) — the stored chain gets a finish line (#4327)
+
+The addendum above makes a legacy row read canonical *forever*, which is the
+correctness guarantee — and, read literally, also a promise that the chain runs
+on that row forever. `os migrate meta --stored`
+(`ObjectStackProtocolImplementation.migrateStoredMetadata`) lets a deployment
+end that for itself: it walks `sys_metadata` (active + draft, all orgs), replays
+the same `applyConversionsToStoredItem` pass, and re-saves each changed body
+through `saveMetaItem` with `source: 'migrate-stored'` — history row, checksum,
+mutation projectors and all. Preview is the default; `--apply` is the only
+writing mode.
+
+- **Not load-bearing, and no flag.** #3855's conclusion stands: an operator-run
+  migration cannot be relied on, so the read path — not this — remains the
+  guarantee, and nothing gates on it having run. Deliberately no `sys_migration`
+  row either: unlike ADR-0104's two gates, a flag here would advertise
+  enforcement that does not exist. The verifiable statement operators wanted is
+  the **re-run** — a second pass reporting every row canonical exits 0, so "my
+  metadata is on protocol N" is a check rather than a belief.
+- **The write path's gate is not bypassed.** A body that still fails the current
+  schema after conversion is refused (422) and reported, exactly as the bullet
+  above describes for reads: it is a genuine contract violation, and the pass
+  has no more standing to persist it than an author does. It keeps reading
+  through the chain and stays fixable in Studio.
+- **The version layer stays verbatim.** `sys_metadata_history` is appended to,
+  never rewritten. Canonicalizing a past version's body would break the
+  checksum↔body pairing this contract depends on — the migration is a new
+  commit, not a rewrite of history.
+- **What the pass does not cover, it names.** Flows (their seam is
+  `AutomationEngine.registerFlow`, which holds the executor registry the
+  conflict guard needs) and types with no repository write path are reported as
+  `skipped` with the reason, never counted as done. Giving flows the same finish
+  line needs a canonicalization entry point on the automation engine — tracked
+  as #4454, and worth doing precisely because the graduated flow-node
+  conversions are where the most stored dialect lives.

packages/cli/package.json

@@ -55,6 +55,7 @@
     "@objectstack/lint": "workspace:*",
     "@objectstack/mcp": "workspace:*",
     "@objectstack/metadata": "workspace:*",
+    "@objectstack/metadata-protocol": "workspace:*",
     "@objectstack/objectql": "workspace:^",
     "@objectstack/observability": "workspace:^",
     "@objectstack/platform-objects": "workspace:*",

packages/cli/src/commands/migrate/meta.stored-flags.test.ts

@@ -0,0 +1,68 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * #4327 — the flag surface where `os migrate meta`'s two modes meet.
+ *
+ * One command, two subjects: `--from N` replays the ADR-0087 chain over an
+ * **author's source** (a config file, no database in reach), `--stored` replays
+ * it over **one deployment's** `sys_metadata` rows (a database, no config in
+ * reach). Everything below pins a boundary that a plausible-looking oclif
+ * declaration gets wrong.
+ */
+import { describe, expect, it } from 'vitest';
+import MigrateMeta, { storedOnlyFlagsIn } from './meta.js';
+
+describe('storedOnlyFlagsIn (#4327)', () => {
+  it('reports the stored-only flags the operator typed', () => {
+    expect(storedOnlyFlagsIn(['--from', '16', '--apply'])).toEqual(['apply']);
+    expect(storedOnlyFlagsIn(['--from=16', '--type=view', '--force'])).toEqual(['force', 'type']);
+    expect(storedOnlyFlagsIn(['--from', '16', '-y'])).toEqual(['yes']);
+  });
+
+  it('says nothing about a run that typed none of them', () => {
+    expect(storedOnlyFlagsIn(['--from', '16', '--step', '--json'])).toEqual([]);
+    expect(storedOnlyFlagsIn([])).toEqual([]);
+  });
+
+  it('never double-reports --yes given both spellings', () => {
+    expect(storedOnlyFlagsIn(['--stored', '--yes', '-y'])).toEqual(['yes']);
+  });
+
+  it('reads argv, not the environment — an exported OS_DATABASE_URL is not a typed flag', () => {
+    // The trap that made `dependsOn` unusable: oclif fills `--database-url`
+    // from `OS_DATABASE_URL`, so a merely-exported env var would have counted
+    // as "provided" and broken `os migrate meta --from N` for anyone who has
+    // one set. Provenance comes from argv precisely so it cannot.
+    expect(storedOnlyFlagsIn(['--from', '16'])).toEqual([]);
+    expect(storedOnlyFlagsIn(['--stored', '--database-url', 'sqlite://x.db'])).toEqual(['database-url']);
+  });
+});
+
+describe('MigrateMeta flag declarations (#4327)', () => {
+  const flags = MigrateMeta.flags as Record<string, any>;
+
+  it('does not declare --from required — that would reject every --stored run', () => {
+    expect(flags.from.required).not.toBe(true);
+  });
+
+  it('makes the authored-chain flags exclusive with --stored', () => {
+    // `--from` names a major an author wrote against; a stored row carries its
+    // own history and gets the full chain regardless. Accepting both would
+    // imply the stored pass honours a range it does not have.
+    for (const name of ['from', 'to', 'step', 'out']) {
+      expect(flags[name].exclusive).toContain('stored');
+    }
+  });
+
+  it('leaves the stored-only flags free of dependsOn', () => {
+    // Enforced by `storedOnlyFlagsIn` instead: see the env-var case above.
+    for (const name of ['apply', 'yes', 'force', 'type', 'database-url']) {
+      expect(flags[name].dependsOn).toBeUndefined();
+    }
+  });
+
+  it('defaults --apply off — preview is the only mode a bare run can have', () => {
+    expect(flags.apply.default).toBe(false);
+    expect(flags.stored.default).toBe(false);
+  });
+});