feat(metadata-protocol/objectql): turn-atomic package publish — a commit cannot half-land (ADR-0067 D2, #3066) (#3074)

* feat(metadata-protocol/objectql): turn-atomic package publish (ADR-0067 D2, #3066)

Phase 1: every draft promotion + the sys_metadata_commit record run inside
ONE engine.transaction() — a mid-batch failure rolls back everything
(publishedCount 0; causal item carries its real error, the rest report
batch_aborted). publishMetaItem is split into promoteDraftForPublish
(guards + promote, metadata-only) and runPublishSideEffects so the batch
can transact promotions while deferring side effects.

Phase 2 (post-commit): registry refresh, DDL, seed batch apply,
materializers, ADR-0094 projections, events — best-effort, surfaced via
materializeApplied.failures instead of faking an unpublish (metadata is
live; boot reconciliation heals side-effect drift).

objectql: engine.transaction() now JOINS an already-open ambient
transaction instead of opening a nested driver transaction — a nested
begin deadlocks single-connection pools (SQLite knex) and escapes the
outer rollback. Locked by two new engine-ambient-transaction tests.

BREAKING (behavioral): consumers relying on partial batch publishes now
get all-or-nothing. Engines without transaction() keep the previous
sequential behavior (same fallback stance as the repo's withTxn).

Tests: publish contract suite rewritten to the two-phase seams (+4 new
cases incl. rollback/commit tracking and side-effect-failure surfacing);
commit-history suite updated; 29 green across the three suites.

* fix(tests): migrate build-probes spy to the phase-1 seam; rescue orphaned search-conformance files

- build-probes.test.ts publish suite: spy promoteDraftForPublish +
  runPublishSideEffects (the batch no longer calls publishMetaItem per
  item, ADR-0067 D2). objectql suite back to 904/904.
- The #3037 packages/dogfood -> packages/qa/dogfood restructure merged
  concurrently with #3065 and did not carry its three just-added files;
  they were orphaned at the old path outside every workspace (silently
  not running in CI). git mv them to packages/qa/dogfood/test — all 5
  dogfood suites green at the new home (21 tests).

* docs(adr-0067): flip to Accepted — Decision-2 turn-atomicity lands with this PR (#3066)

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/turn-atomic-publish.md

@@ -0,0 +1,6 @@
+---
+'@objectstack/metadata-protocol': minor
+'@objectstack/objectql': minor
+---
+
+Package-draft publishing is now turn-atomic (ADR-0067 Decision-2, #3066). `publishPackageDrafts` runs every draft promotion AND the `sys_metadata_commit` record inside ONE engine transaction — a mid-batch failure rolls back the whole batch (`publishedCount: 0`; the causal item carries its real error, the rest report `batch_aborted`). Side effects (registry refresh, table DDL, seed apply, materializers, ADR-0094 projections, events) run after the metadata commits and are surfaced-not-swallowed on failure. `@objectstack/objectql`'s `engine.transaction()` now JOINS an already-open ambient transaction instead of opening a nested driver transaction (deadlock on single-connection pools; escaped the outer rollback). BREAKING (behavioral): API consumers that relied on partial batch publishes ("2 of 3 landed") now get all-or-nothing; engines without `transaction()` (memory driver, minimal stubs) keep the previous sequential behavior.

docs/adr/0067-commit-history-and-rollback-for-ai-authoring.md

@@ -1,6 +1,6 @@
 # ADR-0067: Commit history and rollback for AI authoring — turns become atomic, revertible commits
 
-**Status**: Proposed (2026-06-24) — mostly implemented (2026-07-16 audit): commit grouping (`sys_metadata_commit`), `revertCommit`/`rollbackToPackageCommit`/`listCommits`, REST routes and tests all shipped; **Decision-2 (turn-atomic single-transaction apply, "a commit cannot half-land") is NOT implemented** — publish remains per-item best-effort and commits record partial publishes (`protocol.ts` records over `publishedKeys` after the fact).
+**Status**: Accepted (2026-06-24; completed 2026-07-16) — fully implemented: commit grouping (`sys_metadata_commit`), `revertCommit`/`rollbackToPackageCommit`/`listCommits`, REST routes; **Decision-2 landed via #3066**: `publishPackageDrafts` runs every promotion + the commit record inside ONE `engine.transaction()` (two-phase — side effects post-commit), so a commit cannot half-land; `engine.transaction()` joins ambient transactions to make nested repository writes participate. Locked by `protocol-publish-package-drafts.test.ts` (all-or-nothing + rollback tracking) and `engine-ambient-transaction.test.ts`.
 **Deciders**: ObjectStack Protocol Architects
 **Builds on / amends**: [ADR-0045](./0045-additive-materialization-and-visibility-gate.md) (**amended**: ADR-0045 keeps a *draft + human-confirm* gate on mutations as the safety mechanism; this ADR replaces *confirm-before* with *revert-after* for everything except irreversible data loss, and unifies the two authoring regimes under one primitive — the commit), [ADR-0027](./0027-metadata-authoring-lifecycle.md) (draft workspace — retained as a *review affordance*, demoted from *safety mechanism*), [ADR-0033](./0033-ai-assisted-metadata-authoring.md) ("AI never publishes — it drafts" → **AI commits; commits are revertible**), [ADR-0034](./0034-transactional-writes-and-ambient-transaction.md) (per-write transaction — **extended to span a whole turn**), [ADR-0038](./0038-build-verification-loop.md) (machine gate — runs per commit, before it lands)
 **Consumers**: `@objectstack/objectql` (commit grouping, atomic turn-apply, `revertCommit`, history query — built on the existing `sys_metadata_history` + `restoreVersion`), `@objectstack/runtime` + `@objectstack/rest` (commit/revert routes), `../cloud/service-ai-studio` (turn = commit; auto-commit policy; data-loss confirmation), `../objectui` (commit timeline + "revert to here")

packages/metadata-protocol/src/protocol.ts

@@ -183,7 +183,13 @@ describe('publishPackageDrafts — probes ride the response (ADR-0038 L3)', () =
       get: async (_ref: any, opts: any) =>
         opts?.state === 'draft' ? { body: ITEMS['seed expense_sample'], hash: 'h' } : null,
     });
-    vi.spyOn(protocol, 'publishMetaItem' as never).mockResolvedValue({ success: true, version: 'h', seq: 1 } as never);
+    // ADR-0067 D2 — the batch promotes via the phase-1 seam; side effects are phase 2.
+    vi.spyOn(protocol as any, 'promoteDraftForPublish').mockImplementation(async (req: any) => ({
+      singularType: req.type,
+      orgId: null,
+      result: { version: 'h', seq: 1, item: { body: ITEMS[`${req.type} ${req.name}`] ?? { name: req.name } }, packageId: null },
+    }));
+    vi.spyOn(protocol as any, 'runPublishSideEffects').mockResolvedValue({});
     vi.spyOn(protocol as any, 'applySeedBodies').mockResolvedValue({ success: false, inserted: 0, updated: 0, error: 'boom' });
     // Probe reads: active items + an engine whose table stayed empty.
     (protocol as any).getMetaItem = async ({ type, name }: any) => ({ item: ITEMS[`${type} ${name}`] });

packages/objectql/src/build-probes.test.ts

@@ -11,9 +11,16 @@ import { ObjectQL } from './engine.js';
 
 function makeRecordingDriver() {
   const stores = new Map<string, Map<string, any>>();
-  const seen: { create: Array<{ object: string; transaction: unknown }>; find: Array<{ object: string; transaction: unknown }> } = {
+  const seen: {
+    create: Array<{ object: string; transaction: unknown }>;
+    find: Array<{ object: string; transaction: unknown }>;
+    commit: unknown[];
+    rollback: unknown[];
+  } = {
     create: [],
     find: [],
+    commit: [],
+    rollback: [],
   };
   const storeFor = (o: string) => {
     let s = stores.get(o);
@@ -60,8 +67,8 @@ function makeRecordingDriver() {
     async bulkUpdate() { return []; },
     async bulkDelete() {},
     async beginTransaction() { return { __trx: true, commit: async () => {}, rollback: async () => {} }; },
-    async commit() {},
-    async rollback() {},
+    async commit(trx: unknown) { seen.commit.push(trx); },
+    async rollback(trx: unknown) { seen.rollback.push(trx); },
   };
   return { driver, seen };
 }
@@ -95,4 +102,35 @@ describe('engine ambient transaction (ADR-0034)', () => {
     await engine.insert('thing', { name: 'outside' });
     expect(seen.create.at(-1)!.transaction).toBeUndefined();
   });
+
+  // ADR-0067 D2 — a nested transaction() JOINS the ambient one instead of
+  // opening a second driver transaction (which would deadlock a
+  // single-connection pool and escape the outer rollback). The outer call
+  // owns the one-and-only commit/rollback.
+  it('a nested transaction() joins the ambient transaction (no second begin)', async () => {
+    let outerTrx: unknown;
+    let innerTrx: unknown;
+    await engine.transaction(async (ctx: any) => {
+      outerTrx = ctx.transaction;
+      await engine.transaction(async (innerCtx: any) => {
+        innerTrx = innerCtx.transaction;
+        await engine.insert('thing', { name: 'nested' });
+      });
+    });
+    expect(innerTrx).toBe(outerTrx); // joined, not a fresh begin
+    expect(seen.create[0].transaction).toBe(outerTrx);
+  });
+
+  it('a throw inside a JOINED nested transaction() rolls back the OUTER one', async () => {
+    await expect(engine.transaction(async () => {
+      await engine.insert('thing', { name: 'first' });
+      await engine.transaction(async () => {
+        throw new Error('inner boom');
+      });
+    })).rejects.toThrow('inner boom');
+    // the recording driver saw the write, but the outer tx rolled back —
+    // rollback tracking lives on the driver; assert it was invoked.
+    expect(seen.rollback.length).toBe(1);
+    expect(seen.commit.length).toBe(0);
+  });
 });

packages/objectql/src/engine-ambient-transaction.test.ts

@@ -2950,6 +2950,18 @@ export class ObjectQL implements IDataEngine {
     callback: (trxCtx: any) => Promise<T>,
     baseContext?: any,
   ): Promise<T> {
+    // ADR-0067 D2 — JOIN an already-open ambient transaction instead of
+    // opening a nested driver transaction. A nested begin would acquire a
+    // second connection (a deadlock on single-connection pools like the
+    // SQLite knex pool) and would NOT be covered by the outer rollback —
+    // exactly the half-landing this join prevents: an outer batch
+    // transaction (e.g. `publishPackageDrafts`) must own the one-and-only
+    // commit/rollback for every write made through nested helpers (the
+    // sys-metadata repository's `withTxn`, hook-driven writes, …).
+    const ambient = this.txStore.getStore();
+    if (ambient?.transaction) {
+      return callback({ ...(baseContext ?? {}), transaction: ambient.transaction });
+    }
     const driver = this.defaultDriver ? this.drivers.get(this.defaultDriver) : undefined;
     const drv = driver as any;
     if (!drv?.beginTransaction) {

packages/objectql/src/engine.ts

@@ -171,7 +171,14 @@ describe('ADR-0067 — publishPackageDrafts records a commit', () => {
       listDrafts: async () => [{ type: 'object', name: 'course' }],
       get: async () => null,
     });
-    vi.spyOn(protocol, 'publishMetaItem' as never).mockResolvedValue({ success: true, version: 'h', seq: 7 } as never);
+    // ADR-0067 D2 — the batch promotes via the phase-1 seam (inside one
+    // transaction), not per-item publishMetaItem; side effects are phase 2.
+    vi.spyOn(protocol as any, 'promoteDraftForPublish').mockImplementation(async (req: any) => ({
+      singularType: req.type,
+      orgId: null,
+      result: { version: 'h', seq: 7, item: { body: { name: req.name } }, packageId: null },
+    }));
+    vi.spyOn(protocol as any, 'runPublishSideEffects').mockResolvedValue({});
 
     const res: any = await (protocol as any).publishPackageDrafts({
       packageId: 'app.edu',