diff --git a/.changeset/adr-0104-value-shapes-scan-gate.md b/.changeset/adr-0104-value-shapes-scan-gate.md new file mode 100644 index 0000000000..992ccbb735 --- /dev/null +++ b/.changeset/adr-0104-value-shapes-scan-gate.md @@ -0,0 +1,62 @@ +--- +"@objectstack/spec": minor +"@objectstack/objectql": minor +"@objectstack/cli": minor +--- + +feat(migrate): `os migrate value-shapes` — the per-deployment gate for reference and structured-JSON value shapes (#3438) + +The second of ADR-0104 D1's two evidence gates. Media value shapes already +enforce once a deployment has verified its file migration (#3681); the +reference (`lookup` / `master_detail` / `user` / `tree`) and structured-JSON +(`location` / `address` / `composite` / `repeater` / `record` / `vector`) +classes now get a gate of their own. + +```bash +os migrate value-shapes # scan: reports, writes nothing +os migrate value-shapes --apply # scan + record the deployment flag when clean +``` + +The run walks every stored value of those classes against +`valueSchemaFor(field, 'stored')` — the same predicate the write path enforces, +imported rather than re-derived — and, at zero violations, records +`sys_migration { id: 'adr-0104-value-shapes', verified_at, blocking: 0 }`. +Strict enforcement of these classes reads **that row**, never the platform +version, so upgrading changes nothing until a deployment produces its own +evidence. + +**There is no backfill, deliberately.** The file migration converts legacy +values because the platform narrowed that storage form and owes the conversion. +A malformed `location` is application data whose correct value only its author +knows, so this run reports and prescribes — naming the object, field, type, +count, offending record ids and the parse issue — and the operator fixes and +re-runs. With nothing to convert, `--apply`'s only write is the flag row, which +keeps the #3617 invariant trivially: a dry run changes nothing, and whether a +run changed this deployment's posture never depends on what it found. + +**A separate flag from the file migration**, because it attests a separate +fact. That flag says file values were migrated and their ownership reconciled; +it says nothing about whether a `lookup` id or a `location` payload is well +formed. Gating these classes on it would be borrowing evidence for a fact it +does not cover. + +- New escape hatch **`OS_ALLOW_LAX_VALUE_SHAPES=1`** returns a verified + deployment to warnings, with the same precedence as its media sibling: the + opt-out beats `OS_DATA_VALUE_SHAPE_STRICT_ENABLED`, which beats the flag. + Wrongly staying lenient costs a warning; wrongly enforcing stops a working + app from writing. +- `@objectstack/spec/system` exports `VALUE_SHAPES_MIGRATION_ID`. +- `@objectstack/objectql` exports `scanValueShapes`, `valueShapeScanPassed` + and `formatValueShapeScanReport`. The scanner is read-only and does **not** + record the flag: readers of a migration flag use the spec contract, only + writers depend on `@objectstack/platform-objects`, so the composition lives + with the CLI command rather than inverting the engine's dependencies. +- `validateRecord` gains `valueShapeStrict`, the sibling of + `mediaValueShapeStrict`. Both default to `false`: a caller that cannot say + stays lenient, so nothing starts rejecting merely because the evidence was + unavailable. + +**Nothing changes for an existing deployment until it runs the command.** A +scan that is truncated, or that cannot read an object, fails the gate even with +zero violations found — "none in the part we read" is not the claim the flag +makes. diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index f2beacfda5..e9780c8037 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -628,6 +628,7 @@ where the data lives. | Command | Description | |---------|-------------| | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag | +| `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean | ```bash os migrate files-to-references # Dry run: full report, writes nothing @@ -660,8 +661,9 @@ Exit status is `0` only when the self-check passes, so CI can gate on it. | **Released-file collection** | A field file whose one owning record lets go (the field is cleared or the record deleted) is tombstoned into the declared 30-day grace window; re-referencing the id within the window revives it, and after it the platform sweep reclaims the row and its bytes. Unverified deployments keep every released file forever. | Other value classes are unaffected: a `lookup` or `location` value keeps its own -warn-first rollout, because this migration is evidence about *file* values and -says nothing about theirs. +warn-first rollout until `os migrate value-shapes` (below) supplies *their* +evidence, because this migration is evidence about *file* values and says +nothing about theirs. A dry run writes **nothing** — not the conversions, and not the flag either, @@ -675,6 +677,53 @@ delete check re-reads the flag fresh, so a later failing run stops collection without a restart. +#### `os migrate value-shapes` + +The same gate for the **non-media** value classes — references (`lookup`, +`master_detail`, `user`, `tree`) and structured JSON (`location`, `address`, +`composite`, `repeater`, `record`, `vector`). + +```bash +os migrate value-shapes # Scan: full report, writes nothing +os migrate value-shapes --apply # Scan, then record the flag if clean (prompts) +os migrate value-shapes --apply --yes --json # CI / scripts +os migrate value-shapes --object contact # Restrict to one object (repeatable) +``` + +**This one converts nothing.** Its sibling rewrites legacy file values because +the platform narrowed that storage form and therefore owes the conversion; a +`location` stored as `{latitude, longitude}` instead of `{lat, lng}` is +*application* data whose correct value only its author knows. So the run reports +— object, field, type, how many records, sample record ids, and the parse issue +— and you fix the values (or the code writing them) and re-run until it is +green. Because there is nothing to convert, `--apply`'s only write is the flag +row itself. + +A scan that is **truncated** (by `--max-records`) or that cannot read an object +fails the gate even with zero violations found: "none in the part we read" is +not the claim the flag makes. + +| Once verified | Effect | +| :--- | :--- | +| **Reference + structured-JSON value shapes** | A malformed value of those classes is **rejected** (`400 invalid_type`) instead of warned about. Set `OS_ALLOW_LAX_VALUE_SHAPES=1` to re-open leniency while diagnosing. | + +This flag is deliberately **separate** from the file migration's. That one +attests that file values were migrated and their ownership reconciled — it says +nothing about whether a `lookup` id or a `location` payload is well formed, so +it may not vouch for these classes. A deployment can legitimately have passed +either without the other. + + +`OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1` turns on **every** value class at once, +regardless of which migrations this deployment has run. It is the "I already +know my data" lever, not the route to strictness — the route is running the +migration that produces the evidence. + + +Same writing rules as its sibling: a dry run writes **nothing**, `--apply` is +the only writing mode, a later failing run clears the verified state, and a +running server reads the flag once — **restart it** after a successful apply. + #### A database created by this version needs no migration A deployment whose database the platform **creates from empty** records these @@ -691,8 +740,10 @@ attests nothing and produces its evidence by running the command, because Importing legacy values into such a deployment is rejected at the write path rather than silently accepted. That is the intended outcome; if you must admit -them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` re-opens leniency, and -re-running the migration re-establishes the flag from the data itself. +them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` (media) and +`OS_ALLOW_LAX_VALUE_SHAPES=1` (references and structured JSON) re-open leniency +per class, and re-running the corresponding migration re-establishes its flag +from the data itself. ### Scaffolding diff --git a/content/docs/releases/v17.mdx b/content/docs/releases/v17.mdx index 7a52c0b158..7824fa29e0 100644 --- a/content/docs/releases/v17.mdx +++ b/content/docs/releases/v17.mdx @@ -984,6 +984,23 @@ records** — never the platform version — is what authorises both strict medi value shapes and irreversible file collection. Upgrading changes neither; running the migration does. +**The non-media classes have their own gate** (#3438) — references (`lookup`, +`master_detail`, `user`, `tree`) and structured JSON (`location`, `address`, +`composite`, `repeater`, `record`, `vector`): + +```bash +os migrate value-shapes # scan: reports, writes nothing +os migrate value-shapes --apply # scan, then record the flag if clean +``` + +A separate flag because it attests a separate fact: the file migration says file +values were converted and their ownership reconciled, which tells you nothing +about whether a `lookup` id or a `location` payload is well formed. This one +converts nothing — a malformed `location` is application data only its author +can correct — so it reports the object, field, type, count, sample record ids +and parse issue, and you re-run until it is clean. `OS_ALLOW_LAX_VALUE_SHAPES=1` +re-opens leniency while diagnosing. + **A database created by 17 attests both flags at creation** (#3438), so a new deployment enforces from its first boot instead of waiting for someone to run a migration that, for an empty store, does nothing. The platform attests only a @@ -1306,7 +1323,11 @@ objectui commits on top of the pin 16.1.0 shipped. media value shapes *and* released-file collection for this deployment, so read the report before you `--apply`. Do **not** reach for `OS_DATA_VALUE_SHAPE_STRICT_ENABLED` to get there: it opts every value class - in at once, including ones with no migration behind them. + in at once, regardless of which migrations this deployment has actually run. +- **Reference and structured-JSON values:** run `os migrate value-shapes` and + fix what it reports before `--apply`. It converts nothing — the values it + names are application data — and a scan that was truncated or could not read + an object fails the gate even at zero violations. - **Datasources:** verify every declared datasource connects in every environment — a bound datasource that cannot connect now fails the boot instead of failing every later query. diff --git a/packages/cli/src/commands/migrate/files-to-references.ts b/packages/cli/src/commands/migrate/files-to-references.ts index 09113c745c..0a712f412b 100644 --- a/packages/cli/src/commands/migrate/files-to-references.ts +++ b/packages/cli/src/commands/migrate/files-to-references.ts @@ -17,7 +17,7 @@ import { import { bootSchemaStack } from '../../utils/schema-migrate.js'; import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; import { describeOccupancy } from '../../utils/sqlite-occupancy.js'; -import { resolveStorageCapabilityArg } from '../serve.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; async function confirm(question: string): Promise { if (!process.stdin.isTTY) return false; // non-interactive → require --yes @@ -30,36 +30,6 @@ async function confirm(question: string): Promise { } } -/** - * The settings + storage service plugins, so the booted kernel carries - * `sys_file`/`sys_migration` and the deployment's REAL storage adapter. - * Settings first: the storage plugin re-resolves its adapter from persisted - * settings when a settings service is present, which is how an S3-configured - * deployment's backfill uploads land in S3 rather than on this machine. - * Storage config goes through the SAME resolver `os serve` uses - * (`resolveStorageCapabilityArg`), so the CLI materialises bytes exactly where - * the server would. It did not, and that mattered here more than anywhere: this - * command reconciles what records claim against what storage actually holds, so - * a root that disagrees with the server's reconciles against the wrong tree. - * It built `{ driver: 'local', root }` — keys `StorageServicePluginOptions` does - * not declare — so the adapter silently used the plugin's own `./storage` - * default while the server (since framework#4096) writes under - * `.objectstack/data/uploads`. - */ -async function buildDataMigrationPlugins(): Promise { - const plugins: unknown[] = []; - try { - const { SettingsServicePlugin } = await import('@objectstack/service-settings'); - plugins.push(new SettingsServicePlugin({ registerRoutes: false })); - } catch { - // optional — without it, constructor/env-driven storage config still applies - } - const { StorageServicePlugin } = await import('@objectstack/service-storage'); - const { options } = resolveStorageCapabilityArg(process.env.OS_STORAGE_ROOT); - plugins.push(new StorageServicePlugin({ ...options, registerRoutes: false })); - return plugins; -} - /** * `os migrate files-to-references` — the ADR-0104 D3 data migration, with its * self-check gate (#3617). diff --git a/packages/cli/src/commands/migrate/value-shapes.ts b/packages/cli/src/commands/migrate/value-shapes.ts new file mode 100644 index 0000000000..18c3d9af43 --- /dev/null +++ b/packages/cli/src/commands/migrate/value-shapes.ts @@ -0,0 +1,248 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { Command, Flags } from '@oclif/core'; +import chalk from 'chalk'; +import { createInterface } from 'node:readline'; +import { + printHeader, + printSuccess, + printWarning, + printError, + printInfo, + printStep, + createTimer, + emitJson, +} from '../../utils/format.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; + +async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; // non-interactive → require --yes + const rl = createInterface({ input: process.stdin, output: process.stdout }); + try { + const answer: string = await new Promise((resolve) => rl.question(question, resolve)); + return /^y(es)?$/i.test(answer.trim()); + } finally { + rl.close(); + } +} + +/** + * `os migrate value-shapes` — the ADR-0104 D1 non-media value-shape gate + * (#3438), the sibling of `os migrate files-to-references` (#3617). + * + * Scans every stored reference (`lookup` / `master_detail` / `user` / `tree`) + * and structured-JSON (`location` / `address` / `composite` / `repeater` / + * `record` / `vector`) value against the contract the write path enforces, and + * — on an `--apply` run finding zero violations — records the deployment-level + * `adr-0104-value-shapes` flag. That flag, never the platform version, is what + * turns strict enforcement of those classes on for THIS deployment. + * + * ## No backfill, deliberately + * + * Unlike its sibling this run rewrites nothing. The file migration converts + * legacy values because the platform narrowed that storage form and therefore + * owes the conversion; a malformed `location` is application data whose correct + * value only its author knows. So this reports and prescribes, and the operator + * fixes and re-runs until it is green. + * + * The happy consequence: with nothing to convert, `--apply`'s only write is the + * flag row, so #3617's invariant is trivially preserved — a dry run changes + * nothing, and whether a run changed this deployment's posture never depends on + * what the run found. + * + * ## Why it is not gated on the file migration's flag + * + * That flag attests that file values were migrated and their ownership + * reconciled. It says nothing about whether a `lookup` id or a `location` + * payload is well formed. Reusing it here would be borrowing evidence for a + * fact it does not cover — see the ADR's 2026-07-27 amendment. + */ +export default class MigrateValueShapes extends Command { + static override description = + 'Scan stored reference and structured-JSON field values against the ADR-0104 value contract. ' + + 'Read-only; --apply records the deployment-level migration flag when the scan finds zero violations.'; + + static override examples = [ + '$ os migrate value-shapes', + '$ os migrate value-shapes --apply', + '$ os migrate value-shapes --apply --yes --json', + '$ os migrate value-shapes --object contact --object account', + ]; + + static override flags = { + 'database-url': Flags.string({ + description: 'Database URL to scan (defaults to $OS_DATABASE_URL / the project DB)', + env: 'OS_DATABASE_URL', + }), + apply: Flags.boolean({ + description: + 'Record the deployment migration flag when the scan passes (the scan itself is always read-only)', + default: false, + }), + yes: Flags.boolean({ char: 'y', description: 'Skip the --apply confirmation prompt', default: false }), + object: Flags.string({ + description: 'Restrict to this object (repeatable; default: every object with a covered field)', + multiple: true, + }), + 'max-records': Flags.integer({ + description: + 'Safety bound on records scanned per object — exceeding it truncates the scan and fails the gate', + }), + json: Flags.boolean({ description: 'Output as JSON (implies non-interactive; requires --yes to apply)' }), + }; + + async run(): Promise { + const { flags } = await this.parse(MigrateValueShapes); + const timer = createTimer(); + const apply = flags.apply; + + if (!flags.json) printHeader('Migrate · value-shapes'); + + // No occupancy gate, unlike the file migration: that command rewrites rows, + // so a second writer on the same SQLite file is a real hazard. This one only + // reads. A concurrent writer can still move the counts under us, which the + // scan reports honestly rather than pretending to have frozen the database. + + if (apply && !flags.yes) { + if (flags.json || !process.stdin.isTTY) { + if (flags.json) { + await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); + this.exit(1); + return; + } + printWarning( + 'Apply mode records this deployment\'s migration flag, which turns on strict value-shape ' + + 'enforcement. Re-run with --yes to confirm, or run without --apply to preview.', + ); + this.exit(1); + return; + } + const ok = await confirm( + chalk.bold('\nRecord the value-shape migration flag on this database if the scan passes? [y/N] '), + ); + if (!ok) { + printInfo('Aborted — nothing recorded.'); + return; + } + } + + if (!flags.json) { + printStep(apply ? 'Booting data stack (APPLY mode)…' : 'Booting data stack (scan only)…'); + } + + let stack; + try { + stack = await bootSchemaStack({ + databaseUrl: flags['database-url'], + extraPlugins: await buildDataMigrationPlugins(), + }); + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); } + printError(error.message || String(error)); + this.exit(1); + return; + } + + try { + const engine: any = stack.kernel.getService('objectql'); + if (typeof engine?.getObject !== 'function') { + throw new Error('No ObjectQL engine on this stack — cannot scan.'); + } + // An empty scan is indistinguishable from a clean one, and this command's + // verdict is what authorises strict enforcement — so refuse to run when no + // app metadata is loaded (missing artifact / wrong directory) rather than + // "verify" a database the scan never actually looked at. + const loadedObjects: string[] = + typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : []; + if (!loadedObjects.some((name) => !name.startsWith('sys_'))) { + throw new Error( + 'No app objects are loaded, so the scan would examine nothing. ' + + 'Run "os build" in your project root first (the migration reads dist/objectstack.json), then re-run.', + ); + } + + const { scanValueShapes, valueShapeScanPassed, formatValueShapeScanReport } = + await import('@objectstack/objectql'); + + // In JSON mode keep stdout parseable — route scan warnings to stderr. + const logger = flags.json + ? { info: (m: string) => console.error(m), warn: (m: string) => console.error(m) } + : { info: (m: string) => printInfo(m), warn: (m: string) => printWarning(m) }; + + const report = await scanValueShapes(engine, logger, { + objects: flags.object, + maxRecordsPerObject: flags['max-records'], + }); + const passed = valueShapeScanPassed(report); + + // The flag write is the ONLY write this command makes, and only on an + // apply run that passed. A failing apply run still records — deliberately: + // it stamps `blocking` and clears `verified_at`, so a deployment whose data + // has regressed closes its own gate rather than coasting on an old pass. + let flag: unknown = null; + if (apply) { + const { recordDataMigrationRun } = await import('@objectstack/platform-objects/system'); + const { VALUE_SHAPES_MIGRATION_ID } = await import('@objectstack/spec/system'); + flag = await recordDataMigrationRun(engine, { + migrationId: VALUE_SHAPES_MIGRATION_ID, + passed, + blocking: report.blocking, + advisory: 0, + applied: true, + // Passed as an OBJECT — `recordDataMigrationRun` serialises it. + details: { + scannedObjects: report.scannedObjects.length, + scannedRecords: report.scannedRecords, + fields: report.findings.length, + truncated: report.truncated, + unreadableObjects: report.unreadableObjects, + }, + }); + if (typeof engine.invalidateDataMigrationFlags === 'function') { + engine.invalidateDataMigrationFlags(); + } + } + + if (flags.json) { + await emitJson({ + database: stack.dbLabel, + apply, + scan: report, + gatePassed: passed, + flag, + duration: timer.elapsed(), + }); + if (!passed) this.exit(1); + return; + } + + printInfo(`Database: ${chalk.white(stack.dbLabel)}`); + console.log(''); + console.log(formatValueShapeScanReport(report).join('\n')); + console.log(''); + + if (passed && apply) { + printSuccess( + `Scan clean — recorded the deployment flag. Reference and structured-JSON value shapes ` + + `are now enforced on this deployment (${timer.elapsed()}).`, + ); + } else if (passed) { + printSuccess(`Scan clean (${timer.elapsed()}). Re-run with --apply to record the flag and enforce.`); + } else if (apply) { + printError('Scan found violations — the flag records the failure and the gate stays closed.'); + printInfo('Fix the values named above and re-run; nothing is converted for you.'); + this.exit(1); + } else { + printError('Scan found violations — fix them, then re-run with --apply.'); + this.exit(1); + } + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); } + printError(error.message || String(error)); + this.exit(1); + } finally { + await stack.shutdown(); + } + } +} diff --git a/packages/cli/src/utils/data-migration-plugins.ts b/packages/cli/src/utils/data-migration-plugins.ts new file mode 100644 index 0000000000..d7ca4fbb74 --- /dev/null +++ b/packages/cli/src/utils/data-migration-plugins.ts @@ -0,0 +1,40 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { resolveStorageCapabilityArg } from '../commands/serve.js'; + +/** + * The service plugins a gated data migration boots with, so the kernel carries + * `sys_migration` (the deployment-level flag ledger) — and, for the file + * migration, `sys_file` plus the deployment's REAL storage adapter. + * + * Settings first: the storage plugin re-resolves its adapter from persisted + * settings when a settings service is present, which is how an S3-configured + * deployment's backfill uploads land in S3 rather than on this machine. + * Storage config goes through the SAME resolver `os serve` uses + * (`resolveStorageCapabilityArg`), so the CLI materialises bytes exactly where + * the server would. It did not, and that mattered more here than anywhere: the + * file migration reconciles what records claim against what storage actually + * holds, so a root that disagrees with the server's reconciles against the + * wrong tree. + * + * ⚠️ `sys_migration` is registered by the STORAGE plugin (its first consumer), + * which is why a migration with nothing to do with files still boots it. That + * coupling is not this module's to fix — the ledger is platform infrastructure + * and should not be owned by an optional service — but until it moves, every + * gated migration boots the same set so they all find the same ledger. Tracked + * separately; `os serve` auto-wires storage, so a served deployment always has + * it registered and the runtime gates can read their flags. + */ +export async function buildDataMigrationPlugins(): Promise { + const plugins: unknown[] = []; + try { + const { SettingsServicePlugin } = await import('@objectstack/service-settings'); + plugins.push(new SettingsServicePlugin({ registerRoutes: false })); + } catch { + // optional — without it, constructor/env-driven storage config still applies + } + const { StorageServicePlugin } = await import('@objectstack/service-storage'); + const { options } = resolveStorageCapabilityArg(process.env.OS_STORAGE_ROOT); + plugins.push(new StorageServicePlugin({ ...options, registerRoutes: false })); + return plugins; +} diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index 407362a2bb..72fadfb6bb 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -12,10 +12,11 @@ import { type DroppedFieldsEvent } from '@objectstack/spec/data'; import type { WriteObservabilityOptions } from '@objectstack/spec/contracts'; -import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, FILE_REFERENCE_TYPES, isFileIdToken, RAW_FILE_VALUES_CONTEXT_KEY } from '@objectstack/spec/data'; +import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, FILE_REFERENCE_TYPES, REFERENCE_VALUE_TYPES, STRUCTURED_JSON_TYPES, isFileIdToken, RAW_FILE_VALUES_CONTEXT_KEY } from '@objectstack/spec/data'; import { DATA_MIGRATION_FLAG_OBJECT, FILE_REFERENCES_MIGRATION_ID, + VALUE_SHAPES_MIGRATION_ID, isDataMigrationFlagVerified, } from '@objectstack/spec/system'; import { ExecutionContext, ExecutionContextInput, ExecutionContextSchema } from '@objectstack/spec/kernel'; @@ -2197,6 +2198,30 @@ export class ObjectQL implements IDataEngine { return this.isFileReferencesMigrationVerified(); } + /** + * Does this object declare a reference or structured-JSON field? Same + * per-schema cache and same dormancy rule as {@link objectHasMediaField}: an + * object holding none of these types can hold no violation of them, so it + * must not pay a query to learn that. + */ + private objectHasCoveredValueField(objectSchema: any): boolean { + if (!objectSchema?.fields) return false; + const cached = ObjectQL.coveredValueFieldPresence.get(objectSchema); + if (cached !== undefined) return cached; + const present = Object.values(objectSchema.fields).some( + (def: any) => def && (REFERENCE_VALUE_TYPES.has(def.type) || STRUCTURED_JSON_TYPES.has(def.type)), + ); + ObjectQL.coveredValueFieldPresence.set(objectSchema, present); + return present; + } + private static readonly coveredValueFieldPresence = new WeakMap(); + + /** The reference / structured-JSON value-shape verdict for a write. */ + private async valueShapeStrictFor(objectSchema: any): Promise { + if (!this.objectHasCoveredValueField(objectSchema)) return false; + return this.isValueShapesMigrationVerified(); + } + /** * Has this deployment completed AND verified the ADR-0104 file-as-reference * migration (#3617)? Read once per process from the `sys_migration` flag. @@ -2219,40 +2244,71 @@ export class ObjectQL implements IDataEngine { */ async isFileReferencesMigrationVerified(): Promise { if (!this.fileReferencesMigrationVerified) { - this.fileReferencesMigrationVerified = (async () => { - if (!this._registry.getObject(DATA_MIGRATION_FLAG_OBJECT)) return false; - try { - const rows = await this.find(DATA_MIGRATION_FLAG_OBJECT, { - where: { id: FILE_REFERENCES_MIGRATION_ID }, - limit: 1, - context: { isSystem: true } as ExecutionContextInput, - }); - const row: any = rows?.[0]; - if (!row || row.id !== FILE_REFERENCES_MIGRATION_ID) return false; - const verified = isDataMigrationFlagVerified({ - id: FILE_REFERENCES_MIGRATION_ID, - last_run_at: String(row.last_run_at ?? ''), - verified_at: row.verified_at == null ? null : String(row.verified_at), - // A non-numeric count must read as "not zero", not as 0 — a bad - // coercion lands on NaN, which fails the === 0 test. - blocking: typeof row.blocking === 'number' ? row.blocking : Number(row.blocking ?? Number.NaN), - }); - if (verified) { - this.logger.info( - '[value-shape] this deployment has verified the file-as-reference migration — ' + - 'media value shapes are enforced and released field files may be collected ' + - '(ADR-0104 / #3617)', - ); - } - return verified; - } catch { - return false; // unreadable evidence → stay lenient - } - })(); + this.fileReferencesMigrationVerified = this.readMigrationFlagVerified( + FILE_REFERENCES_MIGRATION_ID, + '[value-shape] this deployment has verified the file-as-reference migration — ' + + 'media value shapes are enforced and released field files may be collected ' + + '(ADR-0104 / #3617)', + ); } return this.fileReferencesMigrationVerified; } + /** + * Has this deployment completed AND verified the ADR-0104 non-media + * value-shape scan (`os migrate value-shapes`, #3438)? Same memoized seam and + * same fail-lenient posture as the file flag above — and a SEPARATE flag, + * because it attests a different fact. The file migration says file values + * were converted and reconciled; it says nothing about whether a `lookup` id + * or a `location` payload is well formed, so it may not vouch for these + * classes. + */ + async isValueShapesMigrationVerified(): Promise { + if (!this.valueShapesMigrationVerified) { + this.valueShapesMigrationVerified = this.readMigrationFlagVerified( + VALUE_SHAPES_MIGRATION_ID, + '[value-shape] this deployment has verified the value-shape scan — reference and ' + + 'structured-JSON value shapes are enforced (ADR-0104 / #3438)', + ); + } + return this.valueShapesMigrationVerified; + } + + /** + * Read one deployment migration flag and answer whether it authorises its + * consumers. Shared by both flags so the "every way of not knowing answers + * false" rule is written once: no `sys_migration` object registered, no row, + * an unreadable table, a malformed row — all `false`. Enforcement derives + * from evidence, and absent evidence is not permission. + * + * Costs nothing on a kernel without the platform objects: the registry + * lookup short-circuits before any query. + */ + private async readMigrationFlagVerified(migrationId: string, verifiedLog: string): Promise { + if (!this._registry.getObject(DATA_MIGRATION_FLAG_OBJECT)) return false; + try { + const rows = await this.find(DATA_MIGRATION_FLAG_OBJECT, { + where: { id: migrationId }, + limit: 1, + context: { isSystem: true } as ExecutionContextInput, + }); + const row: any = rows?.[0]; + if (!row || row.id !== migrationId) return false; + const verified = isDataMigrationFlagVerified({ + id: migrationId, + last_run_at: String(row.last_run_at ?? ''), + verified_at: row.verified_at == null ? null : String(row.verified_at), + // A non-numeric count must read as "not zero", not as 0 — a bad + // coercion lands on NaN, which fails the === 0 test. + blocking: typeof row.blocking === 'number' ? row.blocking : Number(row.blocking ?? Number.NaN), + }); + if (verified) this.logger.info(verifiedLog); + return verified; + } catch { + return false; // unreadable evidence → stay lenient + } + } + /** * Drop the memoized deployment migration flags so the next write re-reads * them. For a host that runs a data migration in-process and wants its @@ -2260,6 +2316,7 @@ export class ObjectQL implements IDataEngine { */ invalidateDataMigrationFlags(): void { this.fileReferencesMigrationVerified = null; + this.valueShapesMigrationVerified = null; } /** @@ -2330,6 +2387,7 @@ export class ObjectQL implements IDataEngine { * {@link invalidateDataMigrationFlags} instead of waiting for a restart. */ private fileReferencesMigrationVerified: Promise | null = null; + private valueShapesMigrationVerified: Promise | null = null; /** Lazily-built index: child object name → roll-up summary descriptors on * parent objects that aggregate it. Invalidated when packages register. */ @@ -3101,6 +3159,7 @@ export class ObjectQL implements IDataEngine { // Resolved once for the whole batch; dormant unless the object declares // a media field, and memoized after the first object that does. const mediaValueShapeStrict = await this.mediaValueShapeStrictFor(schemaForValidation); + const valueShapeStrict = await this.valueShapeStrictFor(schemaForValidation); // Locale + translation hooks for the rejection messages (#3957) — // resolved once for the batch, identical for every row. const msgCtx = this.validationMessageContext(object, opCtx.context); @@ -3108,7 +3167,7 @@ export class ObjectQL implements IDataEngine { if (rowErrors[i] !== undefined) continue; try { normalizeMultiValueFields(schemaForValidation, rows[i]); - validateRecord(schemaForValidation, rows[i], 'insert', { mediaValueShapeStrict, messages: msgCtx }); + validateRecord(schemaForValidation, rows[i], 'insert', { mediaValueShapeStrict, valueShapeStrict, messages: msgCtx }); evaluateValidationRules(schemaForValidation as any, rows[i], 'insert', { logger: this.logger, currentUser: this.buildEvalUser(opCtx.context), skipStateMachine: shouldSkipStateMachine(opCtx.context), messages: msgCtx }); } catch (e) { if (!partialMode) throw e; @@ -3390,11 +3449,12 @@ export class ObjectQL implements IDataEngine { let priorRecord: Record | null = null; const updateSchema = this._registry.getObject(object); const mediaValueShapeStrict = await this.mediaValueShapeStrictFor(updateSchema); + const valueShapeStrict = await this.valueShapeStrictFor(updateSchema); const updateMsgCtx = this.validationMessageContext(object, opCtx.context); if (hookContext.input.id) { await this.encryptSecretFields(object, hookContext.input.data as Record, opCtx.context, hookContext.input.options); normalizeMultiValueFields(updateSchema, hookContext.input.data as Record); - validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, messages: updateMsgCtx }); + validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, valueShapeStrict, messages: updateMsgCtx }); if (needsPriorRecord(updateSchema as any) || (this.hooks.get('afterUpdate')?.length ?? 0) > 0) { const priorAst: QueryAST = { object, where: { id: hookContext.input.id }, limit: 1 }; priorRecord = await driver.findOne(object, priorAst, hookContext.input.options as any); @@ -3420,7 +3480,7 @@ export class ObjectQL implements IDataEngine { } else if (options?.multi && driver.updateMany) { await this.encryptSecretFields(object, hookContext.input.data as Record, opCtx.context, hookContext.input.options); normalizeMultiValueFields(updateSchema, hookContext.input.data as Record); - validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, messages: updateMsgCtx }); + validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, valueShapeStrict, messages: updateMsgCtx }); // [#2982] Consume the middleware-composed AST seeded above, so // the injected row-scoping (RLS write filter, sharing's // editable-rows filter) actually binds the driver operation. Fail diff --git a/packages/objectql/src/index.ts b/packages/objectql/src/index.ts index f3cac898e4..f5c7169b4b 100644 --- a/packages/objectql/src/index.ts +++ b/packages/objectql/src/index.ts @@ -75,6 +75,23 @@ export type { WrapDeclarativeOptions } from './hook-wrappers.js'; // Export Validation export { ValidationError, validateRecord } from './validation/record-validator.js'; export type { FieldValidationError } from './validation/record-validator.js'; +// [ADR-0104 D1 / #3438] The value-shape scan behind `os migrate value-shapes`. +// Read-only by design: it produces the evidence and the CALLER records the +// flag. The engine deliberately does not depend on `@objectstack/platform-objects` +// (readers of a flag use the spec contract; only writers take that dependency), +// so the composition lives with the command, not here. +export { + scanValueShapes, + valueShapeScanPassed, + formatValueShapeScanReport, +} from './validation/scan-value-shapes.js'; +export type { + ValueShapeFinding, + ValueShapeScanReport, + ValueShapeScanEngine, + ValueShapeScanOptions, + ValueShapeScanLogger, +} from './validation/scan-value-shapes.js'; export { evaluateValidationRules, needsPriorRecord, legalNextStates } from './validation/rule-validator.js'; export type { EvaluateRulesOptions } from './validation/rule-validator.js'; export { diff --git a/packages/objectql/src/validation/record-validator.ts b/packages/objectql/src/validation/record-validator.ts index acf9b9d1f6..537221b9e3 100644 --- a/packages/objectql/src/validation/record-validator.ts +++ b/packages/objectql/src/validation/record-validator.ts @@ -352,6 +352,7 @@ function validateOne( skipRequired = false, mediaStrict = false, ctx?: ValidationMessageContext, + valueStrict = false, ): FieldValidationError | null { const fail = ( code: FieldErrorCode, @@ -499,16 +500,23 @@ function validateOne( // ADR-0104 R7 external URLs are the deliberate exception the migration // reports and never converts, so a deployment still holding them fails // its self-check and stays lenient here. - // - **References and structured JSON** stay warn-first: the file migration - // says nothing about whether a `lookup` or `location` value is well - // formed, so gating them on ITS flag would be borrowing evidence for a - // fact it does not cover. They flip when something can vouch for them. + // - **References and structured JSON** enforce on their OWN evidence: the + // `os migrate value-shapes` scan, which walks every stored value of these + // classes against this same schema and records + // `adr-0104-value-shapes` only at zero violations. They are NOT gated on + // the file migration's flag — that one asserts file values were migrated + // and reconciled, and says nothing about whether a `lookup` id or a + // `location` payload is well formed. Borrowing it would be an authority + // answering a question nobody asked it. + // + // Two flags rather than one, because they attest two different facts; a + // deployment can legitimately have passed either without the other. if (REFERENCE_VALUE_TYPES.has(t) || FILE_REFERENCE_TYPES.has(t) || STRUCTURED_JSON_TYPES.has(t)) { const parsed = shapeSchemaFor(def).safeParse(value); if (!parsed.success) { const detail = parsed.error.issues[0]?.message ?? 'invalid value shape'; const isMedia = FILE_REFERENCE_TYPES.has(t); - if (isMedia ? mediaStrictEffective(mediaStrict) : VALUE_SHAPE_STRICT()) { + if (isMedia ? mediaStrictEffective(mediaStrict) : valueShapeStrictEffective(valueStrict)) { return fail('invalid_type', { type: t, detail }, 'invalid_value_shape'); } // The warn-first path is a DEVELOPER log line, not an end-user message — @@ -517,11 +525,9 @@ function validateOne( const message = `${name} has an invalid ${t} value: ${detail}`; warnOnce( `${t}:${name}`, - `[value-shape] ${message} — accepted for now (ADR-0104 warn-first; ` + - (isMedia - ? 'run `os migrate files-to-references --apply` to migrate this deployment and enforce' - : 'set OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1 to enforce') + - ')', + `[value-shape] ${message} — accepted for now (ADR-0104 warn-first; run \`os migrate ` + + (isMedia ? 'files-to-references' : 'value-shapes') + + ' --apply` to migrate this deployment and enforce)', ); } return null; @@ -532,7 +538,12 @@ function validateOne( return null; } -/** Strict value-shape enforcement opt-in (ADR-0104 warn-first rollout). */ +/** + * All-class strict opt-in (ADR-0104). Turns on EVERY value class at once, + * including ones whose per-deployment migration has not run here — so it is + * the "I already know my data" lever, not the blessed route. The blessed route + * is running the migration that produces the evidence. + */ function VALUE_SHAPE_STRICT(): boolean { return typeof process !== 'undefined' && process.env?.OS_DATA_VALUE_SHAPE_STRICT_ENABLED === '1'; } @@ -546,6 +557,16 @@ function LAX_MEDIA_VALUES(): boolean { return typeof process !== 'undefined' && process.env?.OS_ALLOW_LAX_MEDIA_VALUES === '1'; } +/** + * The same escape hatch for the reference / structured-JSON classes, whose + * evidence is the `adr-0104-value-shapes` scan rather than the file migration. + * Deliberately a SECOND variable: the two gates open on different evidence, so + * an operator backing out of one has said nothing about the other. + */ +function LAX_VALUE_SHAPES(): boolean { + return typeof process !== 'undefined' && process.env?.OS_ALLOW_LAX_VALUE_SHAPES === '1'; +} + /** * Does a media value-shape violation reject, for this deployment? * @@ -560,6 +581,57 @@ function mediaStrictEffective(deploymentVerified: boolean): boolean { return deploymentVerified; } +/** + * The reference / structured-JSON counterpart — identical precedence, its own + * pair of inputs. A sibling function rather than one parameterised helper so + * each gate names the evidence it opens on at its own call site; collapsing + * them would save three lines and lose the distinction the ADR spent an + * addendum drawing. + */ +function valueShapeStrictEffective(deploymentVerified: boolean): boolean { + if (LAX_VALUE_SHAPES()) return false; + if (VALUE_SHAPE_STRICT()) return true; + return deploymentVerified; +} + +/** + * Is `value` a violation of `def`'s declared stored shape — the SAME question + * `validateOne` asks, exported so the `os migrate value-shapes` scanner counts + * exactly what strict mode would reject and nothing else. + * + * The scanner must not re-derive this. Two implementations of "malformed" + * drifting by one clause is how a deployment passes a scan and then has writes + * rejected anyway — the migration would be attesting a fact the validator does + * not recognise, which is precisely the borrowed-evidence failure the ADR's + * addendum forbids one layer up. + * + * Returns the first parse issue's message, or `null` when the value conforms. + * A value that is missing (per `isMissing`) is never a violation: absence is + * the `required` check's business, not the shape contract's. + */ +export function valueShapeViolation(def: FieldDef, value: unknown): string | null { + if (isMissing(value)) return null; + const t = def.type; + if (!REFERENCE_VALUE_TYPES.has(t) && !FILE_REFERENCE_TYPES.has(t) && !STRUCTURED_JSON_TYPES.has(t)) { + return null; + } + const parsed = shapeSchemaFor(def).safeParse(value); + if (parsed.success) return null; + return parsed.error.issues[0]?.message ?? 'invalid value shape'; +} + +/** + * Is this field one the value-shape scan covers, and one a client may write? + * `system` / `readonly` / lifecycle columns are skipped for the same reason + * `validateRecord` skips them — the engine owns them, so they are never + * validated on a write and must never be counted as blocking a gate that + * governs writes. + */ +export function isScannableValueShapeField(name: string, def: FieldDef | undefined): boolean { + if (!def || SKIP_FIELDS.has(name) || def.system || def.readonly) return false; + return REFERENCE_VALUE_TYPES.has(def.type) || STRUCTURED_JSON_TYPES.has(def.type); +} + const warnedShapes = new Set(); function warnOnce(key: string, message: string): void { if (warnedShapes.has(key)) return; @@ -598,6 +670,21 @@ export interface ValidateRecordOptions { */ mediaValueShapeStrict?: boolean; + /** + * Has THIS DEPLOYMENT completed and self-check-verified the ADR-0104 + * non-media value-shape scan (`os migrate value-shapes`, #3438)? When true, a + * malformed reference (`lookup` / `master_detail` / `user` / `tree`) or + * structured-JSON (`location` / `address` / `composite` / `repeater` / + * `record` / `vector`) value rejects instead of warning. + * + * Separate from {@link mediaValueShapeStrict} because the two facts are + * attested by two different migrations and either can hold without the other. + * Same default and same reason: a caller that cannot say stays lenient, so + * nothing starts rejecting writes merely because the evidence was + * unavailable. + */ + valueShapeStrict?: boolean; + /** * Locale + translation hooks for the human half of each error (#3957). Omit * and messages render in `en` against the declared labels — the pre-#3957 @@ -624,6 +711,7 @@ export function validateRecord( const errors: FieldValidationError[] = []; const fields = objectSchema.fields; const mediaStrict = options.mediaValueShapeStrict === true; + const valueStrict = options.valueShapeStrict === true; const messages = options.messages; if (mode === 'insert') { @@ -632,7 +720,7 @@ export function validateRecord( for (const [name, def] of Object.entries(fields)) { if (SKIP_FIELDS.has(name)) continue; if (def.system || def.readonly) continue; - const err = validateOne(name, def, data[name], false, mediaStrict, messages); + const err = validateOne(name, def, data[name], false, mediaStrict, messages, valueStrict); if (err) errors.push(err); } } else { @@ -661,7 +749,7 @@ export function validateRecord( // skipRequired: PATCH-omitted fields must not 400. (No def clone — the // registry's own field object flows through so the ADR-0104 value-shape // schema cache, keyed on def identity, hits.) - const err = validateOne(name, def, value, true, mediaStrict, messages); + const err = validateOne(name, def, value, true, mediaStrict, messages, valueStrict); if (err) errors.push(err); } } diff --git a/packages/objectql/src/validation/scan-value-shapes.test.ts b/packages/objectql/src/validation/scan-value-shapes.test.ts new file mode 100644 index 0000000000..59a69c7142 --- /dev/null +++ b/packages/objectql/src/validation/scan-value-shapes.test.ts @@ -0,0 +1,123 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { scanValueShapes, valueShapeScanPassed } from './scan-value-shapes.js'; +import { validateRecord, ValidationError } from './record-validator.js'; + +const OBJECTS: Record = { + contact: { + fields: { + id: { type: 'text' }, + name: { type: 'text' }, + account: { type: 'lookup', reference: 'account' }, + geo: { type: 'location' }, + // Engine-owned: never validated on a write, so never scanned either. + owner_cache: { type: 'lookup', reference: 'sys_user', system: true }, + }, + }, + // No covered field — must not be walked at all. + note: { fields: { id: { type: 'text' }, body: { type: 'textarea' } } }, +}; + +function makeEngine(rows: Record>>) { + const reads: string[] = []; + return { + reads, + getObject: (n: string) => OBJECTS[n], + getConfigs: () => OBJECTS, + async find(object: string) { + reads.push(object); + return rows[object] ?? []; + }, + }; +} + +const silent = { info: () => {}, warn: () => {} }; + +describe('scanValueShapes (ADR-0104 D1 / #3438)', () => { + it('a clean database passes and only walks objects declaring covered fields', async () => { + const engine = makeEngine({ + contact: [{ id: 'c1', account: 'acc_1', geo: { lat: 1, lng: 2 } }], + note: [{ id: 'n1', body: 'hi' }], + }); + const report = await scanValueShapes(engine, silent); + + expect(report.blocking).toBe(0); + expect(valueShapeScanPassed(report)).toBe(true); + expect(report.scannedObjects).toEqual(['contact']); + expect(engine.reads).not.toContain('note'); + }); + + it('counts malformed values per field, with samples and the parse detail', async () => { + const engine = makeEngine({ + contact: [ + { id: 'c1', geo: { latitude: 1, longitude: 2 } }, // the pre-ADR spec shape + { id: 'c2', geo: { latitude: 3, longitude: 4 } }, + { id: 'c3', account: { id: 'acc_1', name: 'Acme' } }, // expanded, not an id + { id: 'c4', account: 'acc_2', geo: { lat: 0, lng: 0 } }, // fine + ], + }); + const report = await scanValueShapes(engine, silent); + + expect(report.scannedRecords).toBe(4); + expect(report.blocking).toBe(3); + const geo = report.findings.find((f) => f.field === 'geo')!; + expect(geo.count).toBe(2); + expect(geo.type).toBe('location'); + expect(geo.sampleRecordIds).toEqual(['c1', 'c2']); + expect(geo.detail).toBeTruthy(); + expect(report.findings.find((f) => f.field === 'account')!.count).toBe(1); + expect(valueShapeScanPassed(report)).toBe(false); + }); + + it('never counts a system/readonly field — the write path never validates one', async () => { + const engine = makeEngine({ + contact: [{ id: 'c1', owner_cache: { id: 'u1', name: 'nope' } }], + }); + const report = await scanValueShapes(engine, silent); + expect(report.blocking).toBe(0); + }); + + it('a missing or null value is not a violation — that is `required`\'s business', async () => { + const engine = makeEngine({ + contact: [{ id: 'c1', account: null, geo: undefined }, { id: 'c2' }], + }); + const report = await scanValueShapes(engine, silent); + expect(report.blocking).toBe(0); + }); + + it('an unreadable object closes the gate even with zero violations found', async () => { + const engine = { + getObject: (n: string) => OBJECTS[n], + getConfigs: () => OBJECTS, + async find(): Promise>> { + throw new Error('table missing'); + }, + }; + const report = await scanValueShapes(engine, silent); + + expect(report.blocking).toBe(0); + expect(report.unreadableObjects).toEqual(['contact']); + expect(report.truncated).toBe(true); + // "none in the part we could read" is not the claim the flag makes. + expect(valueShapeScanPassed(report)).toBe(false); + }); + + it('the scan counts exactly what strict mode rejects — one predicate, not two', async () => { + // The anti-drift property: every value the scan flags must also be a write + // rejection under the strict gate, and every value it passes must write. + const flagged = { geo: { latitude: 1, longitude: 2 } }; + const clean = { geo: { lat: 1, lng: 2 } }; + + const engine = makeEngine({ contact: [{ id: 'c1', ...flagged }] }); + const report = await scanValueShapes(engine, silent); + expect(report.blocking).toBe(1); + + expect(() => + validateRecord(OBJECTS.contact, { ...flagged }, 'update', { valueShapeStrict: true }), + ).toThrow(ValidationError); + expect(() => + validateRecord(OBJECTS.contact, { ...clean }, 'update', { valueShapeStrict: true }), + ).not.toThrow(); + }); +}); diff --git a/packages/objectql/src/validation/scan-value-shapes.ts b/packages/objectql/src/validation/scan-value-shapes.ts new file mode 100644 index 0000000000..79bd6d6cb7 --- /dev/null +++ b/packages/objectql/src/validation/scan-value-shapes.ts @@ -0,0 +1,236 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The ADR-0104 D1 non-media value-shape scan — the evidence half of the + * per-deployment gate `os migrate value-shapes` opens (#3438). + * + * ## Why a scan and not a backfill + * + * Its sibling migration (`os migrate files-to-references`) converts legacy file + * values before reconciling them, because the platform narrowed that storage + * form and therefore owes the conversion. Nothing comparable applies here: a + * `location` stored as `{latitude, longitude}` or a `lookup` holding an + * expanded record object is APPLICATION data whose correct value only its + * author knows. So this run reports and prescribes; it never rewrites. A + * mechanical normaliser is deliberately absent until real reports show a shape + * common enough to justify one — a normaliser that guesses is how + * silently-wrong values got here in the first place. + * + * The consequence is a pleasing simplification: with nothing to convert, + * `--apply`'s only write is the flag row itself, so the #3617 invariant holds + * trivially — a dry run changes nothing, and whether a run changed this + * deployment's posture never depends on what the run happened to find. + * + * ## Why the predicate is imported + * + * `valueShapeViolation` and `isScannableValueShapeField` come from the write- + * path validator. The scan must count exactly what strict mode would reject — + * same type classes, same skip rules, same cached schemas. A second + * implementation of "malformed" drifting by one clause would let a deployment + * pass its scan and still have writes rejected, which is the flag attesting a + * fact the validator does not recognise. + */ + +import { + valueShapeViolation, + isScannableValueShapeField, +} from './record-validator.js'; + +/** One object/field pair holding at least one malformed value. */ +export interface ValueShapeFinding { + object: string; + field: string; + /** The declared field type, so the report can group by what went wrong. */ + type: string; + /** How many records of this object hold a malformed value in this field. */ + count: number; + /** A few offending record ids, so an operator can go and look. */ + sampleRecordIds: string[]; + /** The first parse issue seen — the prescription an author acts on. */ + detail: string; +} + +export interface ValueShapeScanReport { + /** Objects actually walked (those declaring at least one covered field). */ + scannedObjects: string[]; + scannedRecords: number; + /** Object/field pairs with violations. Empty ⇒ the gate may open. */ + findings: ValueShapeFinding[]; + /** Total malformed values across every finding. */ + blocking: number; + /** + * True when a per-object cap or an unreadable object cut the walk short. + * A truncated scan can never open the gate: it is evidence about part of the + * data, and the claim being made is about all of it. + */ + truncated: boolean; + /** Objects that could not be read at all — reported, and they truncate. */ + unreadableObjects: string[]; +} + +export interface ValueShapeScanEngine { + getObject(name: string): any | undefined; + getConfigs?(): Record; + find(object: string, options: Record): Promise>>; +} + +export interface ValueShapeScanOptions { + /** Restrict to these objects (default: every object with a covered field). */ + objects?: string[]; + maxRecordsPerObject?: number; +} + +export interface ValueShapeScanLogger { + info(msg: string, meta?: unknown): void; + warn(msg: string, meta?: unknown): void; +} + +const SCAN_PAGE_SIZE = 500; +const MAX_SAMPLE_IDS = 5; +const SYSTEM_CTX = { isSystem: true } as const; + +/** The covered, client-writable fields of one object. */ +function scannableFieldsOf(engine: ValueShapeScanEngine, objectName: string): Array<{ name: string; def: any }> { + const schema: any = engine.getObject(objectName); + if (!schema?.fields) return []; + return Object.entries(schema.fields) + .filter(([name, def]: [string, any]) => isScannableValueShapeField(name, def)) + .map(([name, def]: [string, any]) => ({ name, def })); +} + +/** + * Walk every stored value of the reference / structured-JSON classes and count + * the ones the write path would reject. + * + * Read-only by construction — this function has no write surface at all, which + * is why the engine type it accepts exposes none. + */ +export async function scanValueShapes( + engine: ValueShapeScanEngine, + logger: ValueShapeScanLogger, + options: ValueShapeScanOptions = {}, +): Promise { + const maxPerObject = options.maxRecordsPerObject ?? 100_000; + let scannedRecords = 0; + let truncated = false; + const unreadableObjects: string[] = []; + + const candidates = + options.objects ?? + (typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : []); + const scannedObjects = candidates.filter((name) => scannableFieldsOf(engine, name).length > 0); + + // Keyed `object.field`, so one finding aggregates every offending row. + const findings = new Map(); + + for (const object of scannedObjects) { + const fields = scannableFieldsOf(engine, object); + let offset = 0; + + for (;;) { + let page: Array>; + try { + page = await engine.find(object, { + fields: ['id', ...fields.map((f) => f.name)], + limit: SCAN_PAGE_SIZE, + offset, + context: { ...SYSTEM_CTX }, + }); + } catch (err) { + // An object we cannot read is not an object we may vouch for. Record it + // AND truncate: the gate's claim is about all the data, so a hole in + // the evidence has to close it. + logger.warn( + `[value-shape] scan: cannot read ${object} (${(err as Error)?.message ?? err}) — ` + + 'reported as unreadable; the gate stays closed', + ); + unreadableObjects.push(object); + truncated = true; + break; + } + if (!page || page.length === 0) break; + + for (const record of page) { + scannedRecords++; + for (const { name, def } of fields) { + const detail = valueShapeViolation(def, record[name]); + if (!detail) continue; + const key = `${object}.${name}`; + const existing = findings.get(key); + const recordId = String(record.id ?? '(unknown id)'); + if (existing) { + existing.count++; + if (existing.sampleRecordIds.length < MAX_SAMPLE_IDS) existing.sampleRecordIds.push(recordId); + } else { + findings.set(key, { + object, + field: name, + type: String(def?.type ?? 'unknown'), + count: 1, + sampleRecordIds: [recordId], + detail, + }); + } + } + } + + offset += page.length; + if (page.length < SCAN_PAGE_SIZE) break; + if (offset >= maxPerObject) { + truncated = true; + break; + } + } + } + + const list = [...findings.values()].sort((a, b) => b.count - a.count); + return { + scannedObjects, + scannedRecords, + findings: list, + blocking: list.reduce((n, f) => n + f.count, 0), + truncated, + unreadableObjects, + }; +} + +/** + * Does this scan authorise the gate? Zero violations AND a complete walk — + * `blocking === 0` on a truncated scan means "none in the part we read", which + * is not the claim the flag makes. + */ +export function valueShapeScanPassed(report: ValueShapeScanReport): boolean { + return report.blocking === 0 && !report.truncated; +} + +/** Human-readable report body, shared by the CLI's dry-run and apply output. */ +export function formatValueShapeScanReport(report: ValueShapeScanReport): string[] { + const lines: string[] = []; + lines.push( + `Scanned ${report.scannedRecords} record(s) across ${report.scannedObjects.length} object(s) ` + + 'for reference / structured-JSON value shapes.', + ); + if (report.findings.length === 0) { + lines.push('✓ No malformed values found.'); + } else { + lines.push(`✗ ${report.blocking} malformed value(s) in ${report.findings.length} field(s):`); + for (const f of report.findings) { + lines.push( + ` • ${f.object}.${f.field} (${f.type}) — ${f.count} record(s): ${f.detail}` + + `\n e.g. ${f.sampleRecordIds.join(', ')}`, + ); + } + lines.push( + 'These are application data, not platform data: fix the values (or the code that writes', + 'them) and re-run. Nothing here is converted for you — the correct value is one only the', + "app's author knows.", + ); + } + if (report.unreadableObjects.length > 0) { + lines.push(`⚠ Unreadable object(s): ${report.unreadableObjects.join(', ')}`); + } + if (report.truncated) { + lines.push('⚠ Scan incomplete — the gate cannot open on partial evidence.'); + } + return lines; +} diff --git a/packages/spec/src/system/migration.zod.ts b/packages/spec/src/system/migration.zod.ts index 1e221d6d54..07454cdd3f 100644 --- a/packages/spec/src/system/migration.zod.ts +++ b/packages/spec/src/system/migration.zod.ts @@ -138,12 +138,24 @@ export const DATA_MIGRATION_FLAG_OBJECT = 'sys_migration'; export const FILE_REFERENCES_MIGRATION_ID = 'adr-0104-file-references'; /** - * Well-known migration id: ADR-0104 D1 value shapes — every stored value of - * the covered non-media classes (single-value references and structured JSON) - * parses against `valueSchemaFor(field, 'stored')`. Unlike the file migration - * there is nothing to convert: a malformed payload is an application-data fix - * only its owner can make, so the scan reports and prescribes, and `--apply`'s - * only write is this flag. Gates strict enforcement of those classes (#3438). + * Well-known migration id: ADR-0104 D1 non-media value shapes — every stored + * reference (`lookup` / `master_detail` / `user` / `tree`) and structured-JSON + * (`location` / `address` / `composite` / `repeater` / `record` / `vector`) + * value scanned against `valueSchemaFor(field, 'stored')` with zero + * violations. Gates strict enforcement of those classes on this deployment + * (#3438). + * + * Deliberately SEPARATE from {@link FILE_REFERENCES_MIGRATION_ID}, which + * asserts a different fact. That flag says this deployment's file-field values + * were migrated and their ownership reconciled; it says nothing about whether a + * `lookup` id or a `location` payload is well formed. Gating these classes on + * it would be borrowing evidence for a fact it does not cover — the same error + * one layer down as an authority answering a question nobody asked (see the + * ADR's 2026-07-27 amendment). + * + * Unlike the file migration this one has no backfill: a malformed `location` is + * an application-data fix only its author can make, so the run reports and + * prescribes, and `--apply`'s only write is the flag row itself. */ export const VALUE_SHAPES_MIGRATION_ID = 'adr-0104-value-shapes';