feat(spec,metadata-protocol): ADR-0118 — transactions reach plugin space through the contract, and atomic stops lying (#4612)

#4612 asks the platform to choose a debt: transactionalize the engine, bless a
shared migration-journal primitive, or rule both out. Scoping it found the
first option's expensive half was paid three majors ago — ADR-0034's ambient
transaction is implemented, and hooks/validation/internal reads already join it
— while the cheap half was never done. So this records ADR-0118, which splits
the ask along where the work actually is.

D1 (here) — `transaction` joins `IObjectQLEngine`. The mechanism ships; only
the declaration was missing, so the metadata protocol's atomic publish, its
`transactionalBatch` probe, and the sys-metadata repository each reached it
through `as unknown as` casts — unchecked claims about a class none of them
import. The contract's own evidence bar was met three times over. Its caveats
(default-driver only; runs without a transaction when the driver lacks
`beginTransaction`) go into the TSDoc as declared meaning, not behaviour to be
discovered. Narrow host surfaces now type their optional member as
`IObjectQLEngine['transaction']` so they cannot drift from it.

D4 (here) — `batchData`'s `atomic` opened no transaction. It broke a loop:
prior writes stayed committed while the response reported them `success: true`
under the one flag guaranteeing they were undone — #4346's class, silent and
destructive. It now runs the batch inside one `engine.transaction()`, and a
failure rolls back everything and says so (`succeeded: 0`, rows marked
ROLLED_BACK / NOT_ATTEMPTED). A runtime that cannot roll back is REFUSED with
501, never degraded — silent degradation is how the flag came to lie.
`BatchOptionsSchema.atomic`'s declared `.default(true)` is aligned down to the
enforced `false`, so opting in is explicit and no existing caller's failure
semantics change underneath them.

D2 (#4617) specifies a framework-owned migration-journal runner: transactions
cannot span a million-row backfill, a whole-DB-snapshot driver, a multi-
datasource write, or a process crash, so the four queued consumers need journal
+ compensation + resume on top of transactions, not instead of them.
D3 (#4618) retires `IDataEngine.batch?` — declared, implemented by nobody,
called by nobody. Caveat hardening is #4619; the sibling fake-atomics in
`deleteManyData`/`updateManyData` are #4620.

Tests: 10 unit pins on the atomic contract (rollback shape, 501 refusal with
zero writes attempted, precedence, non-atomic regression) and 6 integration
pins over a real engine + transacting driver, including that an internal read
during the batch binds to the open transaction — the no-deadlock coverage whose
absence ADR-0034 was written about.

Closes #4612

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKcGqCYCCpMkB5UW8jNPXx

Author: claude

.changeset/adr-0118-plugin-reachable-transactions.md

@@ -0,0 +1,59 @@
+---
+"@objectstack/spec": minor
+"@objectstack/metadata-protocol": minor
+---
+
+feat(spec,metadata-protocol): `IObjectQLEngine.transaction` joins the slot contract, and `batchData`'s `atomic` flag becomes real — rollback or refusal, never silent best-effort (ADR-0118 D1/D4, #4612)
+
+**D1 — the contract fix.** `ObjectQL.transaction()` — ADR-0034's ambient
+transaction, shipped since v8.0.0 — was reachable from plugin space only
+through `as unknown as` casts: the metadata protocol's atomic publish and its
+`transactionalBatch` discovery probe, and the sys-metadata repository's
+`withTxn`, each declared a private structural slice of an engine none of them
+import. It is now declared on `IObjectQLEngine`, required per that contract's
+own rule, with its caveats written into the TSDoc as part of the declared
+meaning rather than left to be discovered: it covers the **default driver
+only**, and when that driver has no `beginTransaction` the callback runs with
+no transaction and no rollback. `MetadataHostEngine` and the sys-metadata
+repository's engine surface now type their optional member as
+`IObjectQLEngine['transaction']`, so a narrow host surface can no longer drift
+from the real signature. Runtime `typeof === 'function'` probes stay — that is
+test-double defence the type system does not replace.
+
+**D4 — the honesty fix.** `batchData`'s `options.atomic` promised "rollback
+entire batch on any failure (transaction mode)" and delivered a `break`
+statement. Every write before the failure stayed committed, and — the part that
+did the real damage — the response reported those rows `success: true` under
+the one flag whose job is to guarantee they were undone.
+
+Now an explicitly atomic batch runs inside ONE `engine.transaction()`: the
+first failure rolls back every prior write, and the response says so
+(`succeeded: 0`, with rows marked `ROLLED_BACK:` / the causal error /
+`NOT_ATTEMPTED:`, and no row reporting success). On a runtime that cannot roll
+back — no `transaction()`, or a default driver without `beginTransaction` — an
+atomic request is **refused** with `501 NOT_IMPLEMENTED` rather than silently
+degrading, matching the cross-object `/batch` route. `atomic` takes precedence
+over `continueOnError`, whose own description already scoped it to
+`atomic=false`. In atomic mode the upsert path no longer falls back to an
+insert when its update throws: inside an aborted transaction that fallback can
+only fail with a secondary error that buries the real cause.
+
+**Aligned declaration.** `BatchOptionsSchema.atomic` declared `.default(true)`
+while no enforcement site delivered atomicity — and the REST route forwards the
+original request body rather than the parsed output, so the declared default
+never reached the loop at all. The default is now `false`: the declaration is
+aligned down to what every site already does, rather than up to what none of
+them did. Honouring the old `true` would have silently flipped the failure
+semantics of every existing batch caller and hard-failed ordinary batches on
+any driver that cannot transact. Callers who were explicitly sending
+`atomic: true` now get what they always asked for; callers sending nothing keep
+today's behaviour exactly.
+
+If you were passing `atomic: true` and relying on partial results surviving a
+failure, that was the bug — switch to `atomic: false` (or omit it) for
+best-effort semantics.
+
+ADR-0118 also rules on two items landing separately: D2 specifies a
+framework-owned migration-journal runner for multi-step migrations too large
+for one transaction, and D3 retires the declared-but-unimplemented
+`IDataEngine.batch?`.

content/docs/references/api/batch.mdx

@@ -86,7 +86,7 @@ const result = BatchConfigSchema.parse(data);
 
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
-| **atomic** | `boolean` | ✅ | If true, rollback entire batch on any failure (transaction mode) |
+| **atomic** | `boolean` | ✅ | Opt-in all-or-nothing. When explicitly true the whole batch runs inside ONE engine transaction: the first failure rolls back every prior write, and the response reports zero successes with rows marked ROLLED_BACK / NOT_ATTEMPTED. A runtime that cannot roll back REFUSES the request (501 NOT_IMPLEMENTED) rather than silently degrading to best-effort — probe `capabilities.transactionalBatch` on /discovery first. Takes precedence over continueOnError. Default false: sequential best-effort. |
 | **returnRecords** | `boolean` | ✅ | If true, return full record data in response |
 | **continueOnError** | `boolean` | ✅ | If true (and atomic=false), continue processing remaining records after errors |
 | **validateOnly** | `any` | optional | [REMOVED] `options.validateOnly` was removed from BatchOptions in @objectstack/spec (#4052). It was never implemented: the batch surfaces persisted regardless, so a "dry-run" would have silently executed. There is no dry-run today — drop the key. If you need to preview a batch without writing, open an issue so it can be designed (no-commit cascade / constraint semantics) and reintroduced as a flag that actually holds. |