Merge remote-tracking branch 'origin/main' into claude/issue-4688-http-request-dual-source

Author: claude

.changeset/data-field-changed-event-removed.md

@@ -0,0 +1,75 @@
+---
+'@objectstack/spec': major
+---
+
+**BREAKING**: `DataEventType` drops `data.field.changed` — it had no producer (ADR-0049 enforce-or-remove, #4673)
+
+`data.field.changed` was declared in the `DataEventType` enum and emitted by
+nothing. The engine's `publishDataEvent` sends `data.record.{created,updated,deleted}`
+and (since #4639) `data.records.{updated,deleted}`; no other producer exists in
+either repository. A subscriber that switched on `data.field.changed` held a
+branch that could never run — and because the surrounding `switch` still
+compiled, nothing ever reported the gap. That is ADR-0078's silently-inert
+declaration, on the event vocabulary.
+
+It also could not have been implemented against this contract as written:
+`DataEventSchema` is record-shaped (`recordId`, `changes`, `before`, `after`)
+with no `field` / `oldValue` / `newValue` slot, so the member advertised a
+granularity the payload has no room for.
+
+**FROM → TO**
+
+| FROM | TO |
+| :--- | :--- |
+| `type: 'data.field.changed'` | `type: 'data.record.updated'`, reading the per-field detail from the payload's `changes` map (with `before` / `after` for surrounding state) |
+
+**The one-line fix** — delete the dead branch and read `changes` off the update
+event:
+
+```ts
+// BEFORE — never ran; no producer ever sent this event
+if (event.type === 'data.field.changed') { onFieldChange(event); }
+
+// AFTER — the changed fields have always ridden on the record event
+if (event.type === 'data.record.updated') {
+  for (const [field, value] of Object.entries(event.changes ?? {})) onFieldChange(field, value);
+}
+```
+
+Removing that branch changes no observable behaviour — it never executed — so
+this is deleting code that could not run, not rebuilding a capability. Note the
+replacement is one event per write rather than N events on a wide table.
+
+**The retirement kit:**
+
+- **Schema** — the member is gone from `DataEventType` (`api/events.zod.ts`),
+  with an in-schema comment recording what was removed and what the live
+  mechanism is. Deliberately **no `retiredKey()` tombstone**: a removed enum
+  VALUE cannot carry a fix-it prescription the way an authorable object key
+  can (the same limit the sharing-rule `full` retirement hit). The enforced
+  channels are `tsc`, which fails any consumer still naming the value in a
+  `DataEventType` position, and the enum parse, which now rejects the name
+  instead of accepting an event that never arrives.
+- **ADR-0087 D3 semantic migration** — `data-field-changed-event-retired` in
+  `migrations/registry.ts` (step 17), carrying the reason and acceptance
+  criteria. Registered as a **semantic TODO rather than a D2 conversion**
+  because this is a runtime EVENT surface: no stack, example or template
+  authors an event name, so there is no source for `os migrate meta` to
+  rewrite. (Webhooks subscribe through the separate authorable
+  `WebhookTriggerType`, whose vocabulary was already trimmed to producers that
+  exist, #3196.)
+- **No liveness-ledger entry** — the ledger governs authorable metadata types
+  (`object`, `field`, `flow`, …); `DataEvent` is a runtime payload contract and
+  has no ledger file. `check:liveness` and `check:empty-state` pass unchanged.
+- **No `authorable-surface.json` movement** — that ratchet tracks authorable
+  *keys* (`api/DataEvent:type` and friends), not enum members, so the key list
+  is unchanged and gates (a)/(b) correctly stay silent.
+- **Tests** — `api/events.test.ts` pins the narrowed `.options`, asserts the
+  retired name no longer parses, and pins the FROM → TO replacement (that
+  `data.record.updated` really does carry `changes` / `before` / `after`).
+- **Docs** — `content/docs/references/api/events.mdx` and
+  `docs/protocol-upgrade-guide.md` regenerated.
+
+If a genuine per-field change stream is ever wanted, it earns its own honest
+contract — the precedent #4639 set for bulk writes — rather than reclaiming
+this slot.

content/docs/references/api/events.mdx

@@ -68,7 +68,7 @@ const result = BulkDataEventSchema.parse(data);
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
 | **id** | `string` | ✅ | Unique event identifier |
-| **type** | `Enum<'data.record.created' \| 'data.record.updated' \| 'data.record.deleted' \| 'data.field.changed'>` | ✅ | Event type |
+| **type** | `Enum<'data.record.created' \| 'data.record.updated' \| 'data.record.deleted'>` | ✅ | Event type |
 | **object** | `string` | ✅ | Object name |
 | **recordId** | `string` | ✅ | Record ID |
 | **changes** | `Record<string, any>` | optional | Changed fields |
@@ -87,7 +87,6 @@ const result = BulkDataEventSchema.parse(data);
 * `data.record.created`
 * `data.record.updated`
 * `data.record.deleted`
-* `data.field.changed`
 
 
 ---

docs/protocol-upgrade-guide.md

@@ -172,6 +172,8 @@ The same window converges the retry policy (#4661). `@objectstack/spec/automatio
 
 The subtle half is the defaults, and it is worth stating because no gate can see it: `job.retryPolicy` defaulted `maxRetries: 3` / `backoffMultiplier: 2` while the automation shape defaulted 0 / 1, and the authorable-surface gate compares KEY SETS — a changed default is invisible to it, to the tombstone mechanism and to `spec_changes` alike. The merged declaration takes 0 / 1 (retry replays side effects, so it is opt-in — the same reading already recorded in `flow-retry-max-retries-required`), and the conversion writes the pre-17 numbers into every existing `job.retryPolicy` that omitted them. Deployed stacks therefore keep their exact behaviour; what changes is only what a NEWLY authored omission means.
 
+The same enforce-or-remove pass reaches the event vocabulary: `DataEventType` drops `data.field.changed` (#4673). It had no producer anywhere — the engine emits `data.record.{created,updated,deleted}` and, since #4639, `data.records.{updated,deleted}` — so a subscriber switching on it held a branch that could never run, and the `switch` still compiled, which is why an empty member could sit in a public enum this long. It could not have been implemented against this contract as written: `DataEventSchema` is record-shaped and has no `field` / `oldValue` / `newValue` slot, so the member advertised a granularity the payload has no room for. Nothing is lost — per-field detail already rides on `data.record.updated` as `changes` (with `before` / `after`), one event per write instead of N on a wide table. Like the driver contract above it is a runtime surface, never stored in stack metadata, so it is one semantic TODO for event consumers rather than a source rewrite, and it carries no tombstone: a removed enum VALUE cannot hold a fix-it error, exactly as the sharing-rule `full` retirement noted. Should a real per-field stream ever be wanted, it earns its own contract on the #4639 precedent rather than reclaiming this slot.
+
 ### Mechanical (applied for you)
 
 | Conversion | Surface | Change | Load window |
@@ -249,6 +251,9 @@ The subtle half is the defaults, and it is worth stating because no gate can see
 - **`data-driver-find-stream-retired`** — `contracts.IDataDriver.findStream / data.DriverInterfaceSchema.findStream` → find() with limit/offset — the paged read whose determinism IS enforced (IDataDriver.find, data/pagination-conformance.ts)
   - Why not automatic: `findStream` was a REQUIRED contract method documented as "optimized for large datasets to avoid memory overflow", and in two of its three implementations it delivered the opposite: `SqlDriver` and `InMemoryDriver` both awaited `find()` for the ENTIRE result set and then yielded it row by row, so the peak memory a caller was promised protection from was already reached before the first yield. The third (`MongoDBDriver._findStream`) did walk a cursor, but it was the one read path in that driver never routed through `buildFindOptions`, so it hardcoded `projection: { _id: 0 }` and silently discarded `query.fields`. None of it was ever observed, because the method had NO caller in either repository: the engine exposes no stream entry, and the REST export, import and bulk-read paths all go through `find()`. The ~20 driver test doubles that existed only to satisfy a required method almost all threw `not implemented`, and nothing ever noticed — which is the proof, not the anecdote. Being REQUIRED, it also taxed every new driver and every test double with an implementation of a capability the platform does not have. Rather than build a caller to justify three implementations, the method is retired; a real cursor-based read should return WITH the caller that needs it (ADR-0049 enforce-or-remove). This is a TS/API contract surface — a driver is CODE, never stack metadata — so there is no source for the chain to rewrite, and deliberately no schema tombstone either: nothing ever ran a driver object through `DriverInterfaceSchema.parse()`, so a prescription there would have no one to reach. The enforced channel is tsc, and it points at callers. ADR-0049 / ADR-0078, #4484.
   - Done when: No code calls `driver.findStream(...)`; large reads page through `find()` with `limit`/`offset` (which guarantees a total order across the whole walk) or go through the export surface. Drivers and test doubles no longer implement the method — one left behind still compiles and is simply never reached, so removing it is cleanup rather than a break, while a CALLER of it no longer type-checks.
+- **`data-field-changed-event-retired`** — `api.DataEventType 'data.field.changed'` → the `data.record.updated` event, whose payload already carries the per-field detail: `changes` (the changed fields), plus `before` / `after`
+  - Why not automatic: `data.field.changed` was declared in `DataEventType` and emitted by nothing — the engine's `publishDataEvent` sends `data.record.{created,updated,deleted}` and (since #4639) `data.records.{updated,deleted}`, and no other producer exists in either repository. A subscriber that switched on it was waiting on an event no producer sends: the branch never ran, and because the surrounding `switch` still compiled, nothing anywhere reported the gap (ADR-0078's silently-inert declaration, on the event vocabulary). `DataEventSchema` could not have carried the semantics even if something had emitted it — the payload is record-shaped (`recordId`, `changes`, `before`, `after`) with no `field` / `oldValue` / `newValue` slot — so the member promised a granularity the contract has no room for. Per-field detail is therefore not lost: it has always ridden on `data.record.updated` as `changes`, which is one event per write rather than N events on a wide table. This is a runtime EVENT surface — no stack, example or template authors an event name (webhooks subscribe through the separate authorable `WebhookTriggerType`, whose vocabulary was already trimmed to producers that exist, #3196) — so there is no source for the chain to rewrite, and deliberately no schema tombstone: a removed ENUM MEMBER cannot carry a retiredKey() fix-it error the way an authorable object key can (the same limit the sharing-rule `full` retirement hit above). The enforced channels are tsc, which fails any consumer still naming the value in a `DataEventType` position, and the enum parse, which now rejects the name instead of accepting an event that never arrives. A genuine per-field stream, if one is ever wanted, gets its own honest contract the way #4639 gave bulk writes theirs. ADR-0049 / ADR-0078, #4673.
+  - Done when: No consumer subscribes to or switches on `data.field.changed`; per-field change detail is read from a `data.record.updated` event's `changes` map (with `before` / `after` for the surrounding state). Deleting the dead branch changes no observable behaviour — it never executed — so the migration is removing code that could not run, not rebuilding a capability.
 
 ---
 

packages/spec/spec-changes.json

@@ -416,6 +416,13 @@
         "migrationId": "data-driver-find-stream-retired",
         "toMajor": 17,
         "rationale": "`findStream` was a REQUIRED contract method documented as \"optimized for large datasets to avoid memory overflow\", and in two of its three implementations it delivered the opposite: `SqlDriver` and `InMemoryDriver` both awaited `find()` for the ENTIRE result set and then yielded it row by row, so the peak memory a caller was promised protection from was already reached before the first yield. The third (`MongoDBDriver._findStream`) did walk a cursor, but it was the one read path in that driver never routed through `buildFindOptions`, so it hardcoded `projection: { _id: 0 }` and silently discarded `query.fields`. None of it was ever observed, because the method had NO caller in either repository: the engine exposes no stream entry, and the REST export, import and bulk-read paths all go through `find()`. The ~20 driver test doubles that existed only to satisfy a required method almost all threw `not implemented`, and nothing ever noticed — which is the proof, not the anecdote. Being REQUIRED, it also taxed every new driver and every test double with an implementation of a capability the platform does not have. Rather than build a caller to justify three implementations, the method is retired; a real cursor-based read should return WITH the caller that needs it (ADR-0049 enforce-or-remove). This is a TS/API contract surface — a driver is CODE, never stack metadata — so there is no source for the chain to rewrite, and deliberately no schema tombstone either: nothing ever ran a driver object through `DriverInterfaceSchema.parse()`, so a prescription there would have no one to reach. The enforced channel is tsc, and it points at callers. ADR-0049 / ADR-0078, #4484."
+      },
+      {
+        "surface": "api.DataEventType 'data.field.changed'",
+        "replacement": "the `data.record.updated` event, whose payload already carries the per-field detail: `changes` (the changed fields), plus `before` / `after`",
+        "migrationId": "data-field-changed-event-retired",
+        "toMajor": 17,
+        "rationale": "`data.field.changed` was declared in `DataEventType` and emitted by nothing — the engine's `publishDataEvent` sends `data.record.{created,updated,deleted}` and (since #4639) `data.records.{updated,deleted}`, and no other producer exists in either repository. A subscriber that switched on it was waiting on an event no producer sends: the branch never ran, and because the surrounding `switch` still compiled, nothing anywhere reported the gap (ADR-0078's silently-inert declaration, on the event vocabulary). `DataEventSchema` could not have carried the semantics even if something had emitted it — the payload is record-shaped (`recordId`, `changes`, `before`, `after`) with no `field` / `oldValue` / `newValue` slot — so the member promised a granularity the contract has no room for. Per-field detail is therefore not lost: it has always ridden on `data.record.updated` as `changes`, which is one event per write rather than N events on a wide table. This is a runtime EVENT surface — no stack, example or template authors an event name (webhooks subscribe through the separate authorable `WebhookTriggerType`, whose vocabulary was already trimmed to producers that exist, #3196) — so there is no source for the chain to rewrite, and deliberately no schema tombstone: a removed ENUM MEMBER cannot carry a retiredKey() fix-it error the way an authorable object key can (the same limit the sharing-rule `full` retirement hit above). The enforced channels are tsc, which fails any consumer still naming the value in a `DataEventType` position, and the enum parse, which now rejects the name instead of accepting an event that never arrives. A genuine per-field stream, if one is ever wanted, gets its own honest contract the way #4639 gave bulk writes theirs. ADR-0049 / ADR-0078, #4673."
       }
     ],
     "removed": []
@@ -891,6 +898,13 @@
           "migrationId": "data-driver-find-stream-retired",
           "toMajor": 17,
           "rationale": "`findStream` was a REQUIRED contract method documented as \"optimized for large datasets to avoid memory overflow\", and in two of its three implementations it delivered the opposite: `SqlDriver` and `InMemoryDriver` both awaited `find()` for the ENTIRE result set and then yielded it row by row, so the peak memory a caller was promised protection from was already reached before the first yield. The third (`MongoDBDriver._findStream`) did walk a cursor, but it was the one read path in that driver never routed through `buildFindOptions`, so it hardcoded `projection: { _id: 0 }` and silently discarded `query.fields`. None of it was ever observed, because the method had NO caller in either repository: the engine exposes no stream entry, and the REST export, import and bulk-read paths all go through `find()`. The ~20 driver test doubles that existed only to satisfy a required method almost all threw `not implemented`, and nothing ever noticed — which is the proof, not the anecdote. Being REQUIRED, it also taxed every new driver and every test double with an implementation of a capability the platform does not have. Rather than build a caller to justify three implementations, the method is retired; a real cursor-based read should return WITH the caller that needs it (ADR-0049 enforce-or-remove). This is a TS/API contract surface — a driver is CODE, never stack metadata — so there is no source for the chain to rewrite, and deliberately no schema tombstone either: nothing ever ran a driver object through `DriverInterfaceSchema.parse()`, so a prescription there would have no one to reach. The enforced channel is tsc, and it points at callers. ADR-0049 / ADR-0078, #4484."
+        },
+        {
+          "surface": "api.DataEventType 'data.field.changed'",
+          "replacement": "the `data.record.updated` event, whose payload already carries the per-field detail: `changes` (the changed fields), plus `before` / `after`",
+          "migrationId": "data-field-changed-event-retired",
+          "toMajor": 17,
+          "rationale": "`data.field.changed` was declared in `DataEventType` and emitted by nothing — the engine's `publishDataEvent` sends `data.record.{created,updated,deleted}` and (since #4639) `data.records.{updated,deleted}`, and no other producer exists in either repository. A subscriber that switched on it was waiting on an event no producer sends: the branch never ran, and because the surrounding `switch` still compiled, nothing anywhere reported the gap (ADR-0078's silently-inert declaration, on the event vocabulary). `DataEventSchema` could not have carried the semantics even if something had emitted it — the payload is record-shaped (`recordId`, `changes`, `before`, `after`) with no `field` / `oldValue` / `newValue` slot — so the member promised a granularity the contract has no room for. Per-field detail is therefore not lost: it has always ridden on `data.record.updated` as `changes`, which is one event per write rather than N events on a wide table. This is a runtime EVENT surface — no stack, example or template authors an event name (webhooks subscribe through the separate authorable `WebhookTriggerType`, whose vocabulary was already trimmed to producers that exist, #3196) — so there is no source for the chain to rewrite, and deliberately no schema tombstone: a removed ENUM MEMBER cannot carry a retiredKey() fix-it error the way an authorable object key can (the same limit the sharing-rule `full` retirement hit above). The enforced channels are tsc, which fails any consumer still naming the value in a `DataEventType` position, and the enum parse, which now rejects the name instead of accepting an event that never arrives. A genuine per-field stream, if one is ever wanted, gets its own honest contract the way #4639 gave bulk writes theirs. ADR-0049 / ADR-0078, #4673."
         }
       ],
       "removed": []