feat(spec,metadata-protocol): ADR-0118 — transactions reach plugin space through the contract, and atomic stops lying (#4612) (#4623)

* feat(spec,metadata-protocol): ADR-0118 — transactions reach plugin space through the contract, and `atomic` stops lying (#4612)

#4612 asks the platform to choose a debt: transactionalize the engine, bless a
shared migration-journal primitive, or rule both out. Scoping it found the
first option's expensive half was paid three majors ago — ADR-0034's ambient
transaction is implemented, and hooks/validation/internal reads already join it
— while the cheap half was never done. So this records ADR-0118, which splits
the ask along where the work actually is.

D1 (here) — `transaction` joins `IObjectQLEngine`. The mechanism ships; only
the declaration was missing, so the metadata protocol's atomic publish, its
`transactionalBatch` probe, and the sys-metadata repository each reached it
through `as unknown as` casts — unchecked claims about a class none of them
import. The contract's own evidence bar was met three times over. Its caveats
(default-driver only; runs without a transaction when the driver lacks
`beginTransaction`) go into the TSDoc as declared meaning, not behaviour to be
discovered. Narrow host surfaces now type their optional member as
`IObjectQLEngine['transaction']` so they cannot drift from it.

D4 (here) — `batchData`'s `atomic` opened no transaction. It broke a loop:
prior writes stayed committed while the response reported them `success: true`
under the one flag guaranteeing they were undone — #4346's class, silent and
destructive. It now runs the batch inside one `engine.transaction()`, and a
failure rolls back everything and says so (`succeeded: 0`, rows marked
ROLLED_BACK / NOT_ATTEMPTED). A runtime that cannot roll back is REFUSED with
501, never degraded — silent degradation is how the flag came to lie.
`BatchOptionsSchema.atomic`'s declared `.default(true)` is aligned down to the
enforced `false`, so opting in is explicit and no existing caller's failure
semantics change underneath them.

D2 (#4617) specifies a framework-owned migration-journal runner: transactions
cannot span a million-row backfill, a whole-DB-snapshot driver, a multi-
datasource write, or a process crash, so the four queued consumers need journal
+ compensation + resume on top of transactions, not instead of them.
D3 (#4618) retires `IDataEngine.batch?` — declared, implemented by nobody,
called by nobody. Caveat hardening is #4619; the sibling fake-atomics in
`deleteManyData`/`updateManyData` are #4620.

Tests: 10 unit pins on the atomic contract (rollback shape, 501 refusal with
zero writes attempted, precedence, non-atomic regression) and 6 integration
pins over a real engine + transacting driver, including that an internal read
during the batch binds to the open transaction — the no-deadlock coverage whose
absence ADR-0034 was written about.

Closes #4612

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKcGqCYCCpMkB5UW8jNPXx

* docs(api): true up the batch `atomic` docs that D4 made wrong (#4612)

The docs-drift check flagged the hand-written API docs, and three of them were
genuinely stale — they documented the bug ADR-0118 D4 fixes:

- `wire-format.mdx` stated it outright: "By default (`options.atomic: true`)
  processing stops at the first failing record — records already written
  earlier in the same batch are **not** rolled back, since there is no wrapping
  database transaction." Accurate before, wrong on both halves now.
- `client-sdk.mdx`'s options table listed `atomic` default `true` with
  "Rollback entire batch on any failure" — the default moved, and the rollback
  claim only became true with this change.
- `data-api.mdx` described `atomic` as the default on `/batch` and `deleteMany`.

Each now describes what the route actually does, including the 501 refusal and
the rolled-back response shape (`succeeded: 0`, ROLLED_BACK / NOT_ATTEMPTED).

Kept honest about scope: `deleteMany` and `updateMany` did NOT get D4's
treatment, so `deleteMany`'s note now says plainly that its `atomic` only stops
the run and rolls nothing back (#4620 tracks the fix). Documenting the fixed
behaviour on routes that still carry the bug would have replaced one inaccuracy
with another.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKcGqCYCCpMkB5UW8jNPXx

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/adr-0118-plugin-reachable-transactions.md

@@ -0,0 +1,59 @@
+---
+"@objectstack/spec": minor
+"@objectstack/metadata-protocol": minor
+---
+
+feat(spec,metadata-protocol): `IObjectQLEngine.transaction` joins the slot contract, and `batchData`'s `atomic` flag becomes real — rollback or refusal, never silent best-effort (ADR-0118 D1/D4, #4612)
+
+**D1 — the contract fix.** `ObjectQL.transaction()` — ADR-0034's ambient
+transaction, shipped since v8.0.0 — was reachable from plugin space only
+through `as unknown as` casts: the metadata protocol's atomic publish and its
+`transactionalBatch` discovery probe, and the sys-metadata repository's
+`withTxn`, each declared a private structural slice of an engine none of them
+import. It is now declared on `IObjectQLEngine`, required per that contract's
+own rule, with its caveats written into the TSDoc as part of the declared
+meaning rather than left to be discovered: it covers the **default driver
+only**, and when that driver has no `beginTransaction` the callback runs with
+no transaction and no rollback. `MetadataHostEngine` and the sys-metadata
+repository's engine surface now type their optional member as
+`IObjectQLEngine['transaction']`, so a narrow host surface can no longer drift
+from the real signature. Runtime `typeof === 'function'` probes stay — that is
+test-double defence the type system does not replace.
+
+**D4 — the honesty fix.** `batchData`'s `options.atomic` promised "rollback
+entire batch on any failure (transaction mode)" and delivered a `break`
+statement. Every write before the failure stayed committed, and — the part that
+did the real damage — the response reported those rows `success: true` under
+the one flag whose job is to guarantee they were undone.
+
+Now an explicitly atomic batch runs inside ONE `engine.transaction()`: the
+first failure rolls back every prior write, and the response says so
+(`succeeded: 0`, with rows marked `ROLLED_BACK:` / the causal error /
+`NOT_ATTEMPTED:`, and no row reporting success). On a runtime that cannot roll
+back — no `transaction()`, or a default driver without `beginTransaction` — an
+atomic request is **refused** with `501 NOT_IMPLEMENTED` rather than silently
+degrading, matching the cross-object `/batch` route. `atomic` takes precedence
+over `continueOnError`, whose own description already scoped it to
+`atomic=false`. In atomic mode the upsert path no longer falls back to an
+insert when its update throws: inside an aborted transaction that fallback can
+only fail with a secondary error that buries the real cause.
+
+**Aligned declaration.** `BatchOptionsSchema.atomic` declared `.default(true)`
+while no enforcement site delivered atomicity — and the REST route forwards the
+original request body rather than the parsed output, so the declared default
+never reached the loop at all. The default is now `false`: the declaration is
+aligned down to what every site already does, rather than up to what none of
+them did. Honouring the old `true` would have silently flipped the failure
+semantics of every existing batch caller and hard-failed ordinary batches on
+any driver that cannot transact. Callers who were explicitly sending
+`atomic: true` now get what they always asked for; callers sending nothing keep
+today's behaviour exactly.
+
+If you were passing `atomic: true` and relying on partial results surviving a
+failure, that was the bug — switch to `atomic: false` (or omit it) for
+best-effort semantics.
+
+ADR-0118 also rules on two items landing separately: D2 specifies a
+framework-owned migration-journal runner for multi-step migrations too large
+for one transaction, and D3 retires the declared-but-unimplemented
+`IDataEngine.batch?`.

content/docs/api/client-sdk.mdx

@@ -504,9 +504,13 @@ The `find` method accepts an options object with **canonical** (recommended) fie
 
 | Property | Type | Default | Description |
 |:---------|:-----|:--------|:------------|
-| `atomic` | `boolean` | `true` | Rollback entire batch on any failure |
+| `atomic` | `boolean` | `false` | Run the batch in one transaction and roll every write back on the first failure. Refused with `501 NOT_IMPLEMENTED` where the driver cannot roll back, rather than degrading to best-effort. Takes precedence over `continueOnError` |
 | `returnRecords` | `boolean` | `false` | Include full records in response |
-| `continueOnError` | `boolean` | `false` | Continue after errors (when atomic=false) |
+| `continueOnError` | `boolean` | `false` | Continue after errors (when atomic is false) |
+
+A rolled-back atomic batch reports `succeeded: 0`, with each row carrying
+`ROLLED_BACK:`, the causal error, or `NOT_ATTEMPTED:` — no row is reported as a
+success, because none of them survived.
 
 ---
 

content/docs/api/data-api.mdx

@@ -228,6 +228,8 @@ Execute a batch operation (create / update / upsert / delete) on multiple record
 
 **Response**: `BatchUpdateResponse` with `succeeded`, `failed`, `total`, and a per-record `results` array. Each entry in `results` has `id`, `success`, an optional `errors` array, and optional `data` (the full record, present when `returnRecords` is `true`).
 
+`options.atomic` defaults to `false` (sequential best-effort, stopping at the first failure). Set it to `true` and the whole batch runs inside one transaction: the first failure rolls back every prior write, and the response reports `succeeded: 0` with each row marked `ROLLED_BACK:`, the causal error, or `NOT_ATTEMPTED:`. A deployment whose driver cannot roll back rejects an atomic request with `501 NOT_IMPLEMENTED` instead of running it best-effort — probe `capabilities.transactionalBatch` on `/discovery` first. `atomic` takes precedence over `continueOnError`.
+
 ### `POST /data/:object/createMany`
 
 Batch create multiple records.
@@ -260,16 +262,20 @@ ignored, on this route and on `deleteMany`.
 
 Batch delete records by ID list.
 
-**Body**: `{ "ids": ["1", "2", "3"], "options": { "atomic": true } }` — `options` is
+**Body**: `{ "ids": ["1", "2", "3"], "options": { "continueOnError": true } }` — `options` is
 the same `BatchOptions` bag `/batch` takes. The body is validated against the
 contract and unknown keys are dropped: the id list is the *only* thing that
 selects rows, so no body key can widen the delete into a filter.
 
 **Response**: `BatchUpdateResponse` — one `results` entry per id. Records are
 deleted one at a time by primary key, so each honours `deleteBehavior`
-(`cascade` / `set_null` / `restrict`) on relations pointing at it. `atomic`
-(default) stops the run at the first failure; `atomic: false` with
-`continueOnError: true` processes the remaining ids and reports the failures.
+(`cascade` / `set_null` / `restrict`) on relations pointing at it. The run stops
+at the first failure; `continueOnError: true` processes the remaining ids and
+reports the failures instead.
+
+Note that on this route `atomic` only stops the run — deletes already performed
+are **not** rolled back. Unlike `/batch`, `deleteMany` has no wrapping
+transaction yet.
 
 ### Batch size
 

content/docs/api/wire-format.mdx

@@ -471,7 +471,11 @@ release group); any code reading them directly should move to `error.code`.
 
 **`POST /api/v1/data/task/batch`**
 
-Process many records of a **single** operation type in one request. The body carries one `operation` (`create`, `update`, `upsert`, or `delete`) plus a `records` array. By default (`options.atomic: true`) processing stops at the first failing record — records already written earlier in the same batch are **not** rolled back, since there is no wrapping database transaction. Set `options.atomic: false` (with `options.continueOnError: true`) to keep processing every record and collect a full partial-success report.
+Process many records of a **single** operation type in one request. The body carries one `operation` (`create`, `update`, `upsert`, or `delete`) plus a `records` array.
+
+By default (`options.atomic` omitted or `false`) processing stops at the first failing record, and records written earlier in the same batch are **not** rolled back — there is no wrapping transaction. Add `options.continueOnError: true` to keep going instead and collect a full partial-success report.
+
+Send `options.atomic: true` to run the whole batch inside one database transaction: the first failure rolls back every prior write, and the response reports zero successes with each row marked `ROLLED_BACK:`, the causal error, or `NOT_ATTEMPTED:`. A deployment whose driver cannot roll back **rejects** an atomic request with `501 NOT_IMPLEMENTED` rather than quietly running it best-effort — probe `capabilities.transactionalBatch` on `/discovery` to know in advance. `atomic` takes precedence over `continueOnError`.
 
 ### Request
 
@@ -483,7 +487,7 @@ Process many records of a **single** operation type in one request. The body car
     { "id": "tsk_01HQ4B8C0E4G6H9K3L5M", "data": { "status": "done" } }
   ],
   "options": {
-    "atomic": true,
+    "atomic": false,
     "continueOnError": false
   }
 }
@@ -509,7 +513,7 @@ The response is the `BatchUpdateResponse` envelope: a top-level `success` flag p
 
 ### Partial Failure Response
 
-When `options.atomic: false` and some records fail, the failing entries carry a single `error` message string (not an array):
+When the batch is not atomic and some records fail, the failing entries carry a single `error` message string (not an array). An atomic batch never returns this shape — it either commits everything or reports every row as failed:
 
 ```json
 {

content/docs/references/api/batch.mdx

@@ -86,7 +86,7 @@ const result = BatchConfigSchema.parse(data);
 
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
-| **atomic** | `boolean` | ✅ | If true, rollback entire batch on any failure (transaction mode) |
+| **atomic** | `boolean` | ✅ | Opt-in all-or-nothing. When explicitly true the whole batch runs inside ONE engine transaction: the first failure rolls back every prior write, and the response reports zero successes with rows marked ROLLED_BACK / NOT_ATTEMPTED. A runtime that cannot roll back REFUSES the request (501 NOT_IMPLEMENTED) rather than silently degrading to best-effort — probe `capabilities.transactionalBatch` on /discovery first. Takes precedence over continueOnError. Default false: sequential best-effort. |
 | **returnRecords** | `boolean` | ✅ | If true, return full record data in response |
 | **continueOnError** | `boolean` | ✅ | If true (and atomic=false), continue processing remaining records after errors |
 | **validateOnly** | `any` | optional | [REMOVED] `options.validateOnly` was removed from BatchOptions in @objectstack/spec (#4052). It was never implemented: the batch surfaces persisted regardless, so a "dry-run" would have silently executed. There is no dry-run today — drop the key. If you need to preview a batch without writing, open an issue so it can be designed (no-commit cascade / constraint semantics) and reintroduced as a flag that actually holds. |