feat(spec): field runtime value-shape contract — ADR-0104 phase 1 (D1)

Spec now owns the runtime value shape of every field type
(data/field-value.zod.ts): semantic type classes, the shared
isMultiValueField, and valueSchemaFor(field, 'stored' | 'expanded').

Consumers converged (each loses its private hand-copied type lists):
- objectql record-validator: derives from spec; previously-opaque types
  (single references, file-likes, location/address/composite/repeater/
  record/vector) get warn-first shape checks (strict via
  OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1); update path no longer clones
  field defs so the per-def schema cache hits.
- rest import-coerce: six local sets → spec-derived.
- driver-sql: JSON_COLUMN_TYPES / NUMERIC_SCALAR_TYPES membership now
  spec-derived (+ driver-internal aliases).
- qa/dogfood: field-zoo MATRIX extracted to field-zoo.matrix.ts; new
  field-zoo-value-shape.test.ts pins contract ⇔ oracle (45 vectors).

Deprecated (removal rides next spec major; FROM→TO in changeset):
CurrencyValueSchema (currency IS a bare number), LocationCoordinatesSchema
(stored shape is {lat, lng} → LocationValueSchema). AddressSchema adopted
as the enforced address value contract.

Tests: spec 6834, objectql 1039, rest 331, driver-sql 284, dogfood
value-shape 45 — all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHpGw3GBA9aFpfwVArRWfd

Author: claude

.changeset/adr-0104-d1-value-shape-contract.md

@@ -0,0 +1,41 @@
+---
+"@objectstack/spec": minor
+"@objectstack/objectql": minor
+"@objectstack/rest": patch
+"@objectstack/driver-sql": patch
+---
+
+feat(spec): field runtime value-shape contract — ADR-0104 phase 1 (D1)
+
+`@objectstack/spec/data` now owns the runtime VALUE shape of every field type
+(`field-value.zod.ts`): semantic type classes (`STRING_VALUE_TYPES`,
+`NUMERIC_VALUE_TYPES`, `REFERENCE_VALUE_TYPES`, `FILE_REFERENCE_TYPES`,
+`STRUCTURED_JSON_TYPES`, `MULTI_CAPABLE_TYPES`, …), the shared
+`isMultiValueField`, and `valueSchemaFor(field, 'stored' | 'expanded')`. The
+four consumers that each hand-copied this knowledge (objectql record-validator,
+rest import-coerce, driver-sql column classification, qa conformance) now
+derive from the spec, and the field-zoo round-trip MATRIX is asserted against
+the contract so the two cannot drift.
+
+**Write-path change (objectql, warn-first):** previously-unvalidated types —
+single `lookup`/`master_detail`/`user`/`tree`, `file`/`image`/`avatar`/
+`video`/`audio`, `location`, `address`, `composite`, `repeater`, `record`,
+`vector` — are now checked against the contract. A violation **logs a warning
+and passes** in this release (legacy rows must not strand their records);
+set `OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1` to enforce as a
+`400 VALIDATION_FAILED`. The flip to strict-by-default rides a later minor
+(ADR-0104 R1/R2).
+
+**Deprecations (removal rides the next spec major), FROM → TO:**
+
+- `CurrencyValueSchema` (`{value, currency}`) → none. A `currency` field's
+  value is a **bare number** everywhere in the runtime (validator, SQL `float`
+  column, import coercion, field-zoo oracle); the currency code lives in field
+  config. Use `valueSchemaFor({type: 'currency'})`.
+- `LocationCoordinatesSchema` (`{latitude, longitude}`) → `LocationValueSchema`
+  (`{lat, lng}`) — the shape the platform actually stores.
+- `AddressSchema` is **adopted** (unchanged) as the enforced `address` value
+  contract via `AddressValueSchema`.
+
+No stored data changes shape; the contract codifies deployed reality
+("reality wins", ADR-0104 D1).

packages/objectql/src/validation/record-validator.test.ts

@@ -345,3 +345,63 @@ describe('validateRecord — url field accepts relative + inline URLs', () => {
     expect(() => validateRecord(schema, { image: 'notaurl' }, 'update')).toThrow(/valid URL/i);
   });
 });
+
+/**
+ * ADR-0104 D1 — value-shape contract for previously-opaque types.
+ *
+ * Warn-first rollout: a shape violation on reference/file/structured-JSON
+ * types logs (once per field) and passes; `OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1`
+ * turns it into a normal invalid_type rejection.
+ */
+describe('validateRecord — ADR-0104 value shapes (warn-first / strict)', () => {
+  const schema = {
+    fields: {
+      account: { type: 'lookup', reference: 'accounts' },
+      geo: { type: 'location' },
+      doc: { type: 'file' },
+      dims: { type: 'vector' },
+    },
+  };
+
+  const withStrict = (fn: () => void) => {
+    process.env.OS_DATA_VALUE_SHAPE_STRICT_ENABLED = '1';
+    try { fn(); } finally { delete process.env.OS_DATA_VALUE_SHAPE_STRICT_ENABLED; }
+  };
+
+  it('accepts contract-conformant values in both modes', () => {
+    const data = {
+      account: 'acc_0001',
+      geo: { lat: 37.77, lng: -122.42 },
+      doc: { url: 'https://cdn/f.pdf', name: 'f.pdf', size: 1024 },
+      dims: [0.1, 0.2],
+    };
+    expect(() => validateRecord(schema, { ...data }, 'update')).not.toThrow();
+    withStrict(() => expect(() => validateRecord(schema, { ...data }, 'update')).not.toThrow());
+  });
+
+  it('warn-first: malformed shapes pass by default (legacy rows must not strand records)', () => {
+    expect(() => validateRecord(schema, { geo: { latitude: 1, longitude: 2 } }, 'update')).not.toThrow();
+    expect(() => validateRecord(schema, { account: { id: 'acc_1' } }, 'update')).not.toThrow();
+  });
+
+  it('strict: malformed shapes reject with invalid_type', () => {
+    withStrict(() => {
+      try {
+        validateRecord(schema, { geo: { latitude: 37.77, longitude: -122.42 } }, 'update');
+        expect.unreachable('expected ValidationError');
+      } catch (e) {
+        expect(e).toBeInstanceOf(ValidationError);
+        const err = e as ValidationError;
+        expect(err.fields[0]?.field).toBe('geo');
+        expect(err.fields[0]?.code).toBe('invalid_type');
+      }
+      // expanded-form object at a stored-form position (unexpanded write)
+      expect(() => validateRecord(schema, { account: { id: 'acc_1', name: 'Acme' } }, 'update')).toThrow(ValidationError);
+      // scalar at a vector
+      expect(() => validateRecord(schema, { dims: 'not-a-vector' }, 'update')).toThrow(ValidationError);
+      // file: id/url string AND inline object both remain legal pre-D3
+      expect(() => validateRecord(schema, { doc: 'file_01HXYZ' }, 'update')).not.toThrow();
+      expect(() => validateRecord(schema, { doc: 42 }, 'update')).toThrow(ValidationError);
+    });
+  });
+});

packages/objectql/src/validation/record-validator.ts

@@ -31,6 +31,14 @@
  * the UI can highlight the specific input.
  */
 
+import {
+  isMultiValueField as specIsMultiValueField,
+  valueSchemaFor,
+  REFERENCE_VALUE_TYPES,
+  FILE_REFERENCE_TYPES,
+  STRUCTURED_JSON_TYPES,
+} from '@objectstack/spec/data';
+
 // Lifecycle columns the engine always owns and the client never supplies. These
 // are skipped by NAME because they are not author-declared business fields.
 // NOTE: `organization_id` / `tenant_id` are intentionally NOT here (#1592) — the
@@ -145,17 +153,11 @@ function optionValues(options: FieldDef['options']): string[] {
 /**
  * A field whose persisted value is an ARRAY of scalars: either an
  * inherently-multi type, or a single-value type flagged `multiple: true`.
- * Per the spec (field.zod.ts), `multiple` applies to select/lookup/file/image;
- * `radio` shares the select branch and `user` is stored identically to
- * `lookup` (FK column, `multiple` ⇒ JSON array) — the runtime expands
- * `Field.user` with `type: 'user'`, so it must be recognized here too.
+ * THE definition is the spec's (ADR-0104 D1) — previously a hand-copy here
+ * and in rest/import-coerce that had to be kept in lock-step manually.
  */
-const MULTI_CAPABLE_TYPES = new Set(['select', 'radio', 'lookup', 'user', 'file', 'image']);
-
 function isMultiValueField(def: FieldDef): boolean {
-  const t = def.type;
-  if (t === 'multiselect' || t === 'checkboxes' || t === 'tags') return true;
-  return MULTI_CAPABLE_TYPES.has(t as string) && def.multiple === true;
+  return specIsMultiValueField(def as { type: string; multiple?: boolean });
 }
 
 /**
@@ -223,12 +225,12 @@ export function coerceBooleanFields<T extends Record<string, unknown>>(
   return (copy ?? row) as T;
 }
 
-function validateOne(name: string, def: FieldDef, value: unknown): FieldValidationError | null {
+function validateOne(name: string, def: FieldDef, value: unknown, skipRequired = false): FieldValidationError | null {
   // ── required ────────────────────────────────────────────────────
   // `autonumber` is runtime-owned: the value is generated by the engine /
   // driver (the SQL driver assigns it from a persistent sequence AFTER this
   // validation runs), so a missing value is never a client error — see #1603.
-  if (def.required && isMissing(value) && def.type !== 'autonumber') {
+  if (!skipRequired && def.required && isMissing(value) && def.type !== 'autonumber') {
     return { field: name, code: 'required', message: `${name} is required` };
   }
   if (isMissing(value)) return null; // nothing else to check
@@ -338,12 +340,62 @@ function validateOne(name: string, def: FieldDef, value: unknown): FieldValidati
     return null;
   }
 
-  // Other types (lookup, file, formula, json, location, etc.) — no
-  // strict shape check at this layer; reference integrity is handled
-  // elsewhere (lookup) and the rest are opaque payloads.
+  // ── previously-opaque types: value-shape contract (ADR-0104 D1) ─────
+  // Single-value references (id string), file-likes (id/url string or the
+  // pre-D3 inline metadata object), and structured JSON payloads
+  // (location/address/composite/repeater/record/vector) are checked against
+  // the spec's `valueSchemaFor`. Warn-first rollout (ADR-0104 R1/R2): a
+  // violation logs instead of rejecting, so legacy rows written under the
+  // lax regime don't strand their records; set
+  // `OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1` to enforce as a 400. The flip to
+  // error-by-default rides a later minor once telemetry is quiet.
+  if (REFERENCE_VALUE_TYPES.has(t) || FILE_REFERENCE_TYPES.has(t) || STRUCTURED_JSON_TYPES.has(t)) {
+    const parsed = shapeSchemaFor(def).safeParse(value);
+    if (!parsed.success) {
+      const detail = parsed.error.issues[0]?.message ?? 'invalid value shape';
+      const message = `${name} has an invalid ${t} value: ${detail}`;
+      if (VALUE_SHAPE_STRICT()) {
+        return { field: name, code: 'invalid_type', message };
+      }
+      warnOnce(`${t}:${name}`, `[value-shape] ${message} — accepted for now (ADR-0104 warn-first; set OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1 to enforce)`);
+    }
+    return null;
+  }
+
+  // Remaining types (formula/summary/autonumber outputs, json/code payloads)
+  // are explicitly open per the spec contract — see field-value.zod.ts.
   return null;
 }
 
+/** Strict value-shape enforcement opt-in (ADR-0104 warn-first rollout). */
+function VALUE_SHAPE_STRICT(): boolean {
+  return typeof process !== 'undefined' && process.env?.OS_DATA_VALUE_SHAPE_STRICT_ENABLED === '1';
+}
+
+const warnedShapes = new Set<string>();
+function warnOnce(key: string, message: string): void {
+  if (warnedShapes.has(key)) return;
+  warnedShapes.add(key);
+  console.warn(message);
+}
+
+/**
+ * Per-field-definition cache of the spec's derived value schema. Building a
+ * Zod schema is an order of magnitude costlier than parsing with it, so the
+ * derivation runs once per field def (ADR-0104 performance budget). Keyed on
+ * def identity — `validateRecord`'s update path passes the registry's own
+ * field objects (no clones), so the map hits.
+ */
+const shapeSchemaCache = new WeakMap<FieldDef, ReturnType<typeof valueSchemaFor>>();
+function shapeSchemaFor(def: FieldDef): ReturnType<typeof valueSchemaFor> {
+  let schema = shapeSchemaCache.get(def);
+  if (!schema) {
+    schema = valueSchemaFor(def as { type: string; multiple?: boolean; options?: FieldDef['options'] }, 'stored');
+    shapeSchemaCache.set(def, schema);
+  }
+  return schema;
+}
+
 /**
  * Validate a payload against a list of declared fields. `objectSchema`
  * comes from `ObjectQL.getRegistry().getObject(name)` and exposes a
@@ -377,8 +429,10 @@ export function validateRecord(
       const def = fields[name];
       if (!def) continue;
       if (def.system || def.readonly) continue;
-      // Clone def with required=false so PATCH-omitted-fields don't 400.
-      const err = validateOne(name, { ...def, required: false }, value);
+      // skipRequired: PATCH-omitted fields must not 400. (No def clone — the
+      // registry's own field object flows through so the ADR-0104 value-shape
+      // schema cache, keyed on def identity, hits.)
+      const err = validateOne(name, def, value, true);
       if (err) errors.push(err);
     }
   }

packages/plugins/driver-sql/src/sql-driver.ts

@@ -9,6 +9,7 @@
 
 import type { QueryAST, DriverOptions, SchemaMode } from '@objectstack/spec/data';
 import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, type AutonumberToken } from '@objectstack/spec/data';
+import { STRUCTURED_JSON_TYPES, FILE_REFERENCE_TYPES, MULTI_OPTION_TYPES, NUMERIC_VALUE_TYPES } from '@objectstack/spec/data';
 import type { IDataDriver } from '@objectstack/spec/contracts';
 import { StorageNameMapping } from '@objectstack/spec/system';
 import { ExternalSchemaModeViolationError } from '@objectstack/spec/shared';
@@ -57,10 +58,14 @@ const GLOBAL_TENANT = '__global__';
  * rest are arrays.
  */
 const JSON_COLUMN_TYPES = new Set<string>([
-  'json', 'object', 'array', 'record',
-  'image', 'file', 'avatar', 'video', 'audio',
-  'location', 'address', 'composite',
-  'multiselect', 'checkboxes', 'tags', 'repeater', 'vector',
+  // Spec value-shape classes (ADR-0104 D1): structured JSON payloads, the
+  // (pre-D3) inline file metadata objects, and the inherently-array option
+  // types. Membership is owned by @objectstack/spec — a type added there
+  // becomes a JSON column here without touching this file.
+  ...STRUCTURED_JSON_TYPES, ...FILE_REFERENCE_TYPES, ...MULTI_OPTION_TYPES,
+  // Driver-internal aliases (external/introspected columns) — not authorable
+  // FieldTypes, so they stay a local extra.
+  'object', 'array',
 ]);
 
 /**
@@ -79,9 +84,9 @@ const JSON_COLUMN_TYPES = new Set<string>([
  * gap for the numeric scalars.
  */
 const NUMERIC_SCALAR_TYPES = new Set<string>([
-  'integer', 'int',
-  'float', 'number', 'currency', 'percent', 'summary',
-  'rating', 'slider', 'progress',
+  // Spec numeric value class (ADR-0104 D1) + driver-internal SQL aliases.
+  ...NUMERIC_VALUE_TYPES,
+  'integer', 'int', 'float',
 ]);
 
 /**