feat(engine): Phase 11.4 BEGIN CONCURRENT writes + commit-time validation (SQLR-22) (#125)

The headline slice of Phase 11. Multiple sibling Connections can
each hold their own open BEGIN CONCURRENT transaction; commits
validate against MvStore and abort with SQLRiteError::Busy on
row-level write-write conflict. The four plan-required tests
pass: disjoint inserts both commit, same-row updates collide and
one wins, aborted writes never become visible, retry-after-Busy
succeeds.

New `sqlrite::mvcc::transaction::ConcurrentTx` (per-Connection):
- TxHandle (RAII registry entry, drops at COMMIT/ROLLBACK)
- tables: HashMap<String, Table> — working state, swapped with
  db.tables for the duration of each statement's executor pass
- tables_at_begin: HashMap<String, Table> — immutable BEGIN-time
  clone, used at COMMIT to derive the write-set without seeing
  other transactions' commits as bogus DELETEs
- schema_at_begin: Vec<String> — sorted table-name fingerprint

Connection wiring:
- New concurrent_tx: Option<ConcurrentTx> field — per-handle so
  N siblings can each carry their own open tx.
- Connection::execute pre-parses for BEGIN CONCURRENT / COMMIT /
  ROLLBACK before sqlparser runs (sqlparser 0.61 doesn't have a
  Concurrent modifier; same intercept pattern as PRAGMA).
- begin_concurrent: pre-conditions (journal_mode = mvcc, no
  active tx, not read-only) → ConcurrentTx::begin → store.
- execute_in_concurrent_tx: rejects DDL via string-prefix check;
  std::mem::swap(db.tables, tx.tables); parks dummy TxnSnapshot
  on db.txn to suppress auto-save; runs process_command;
  unwinds in reverse. Executor itself unchanged.
- commit_concurrent: schema_unchanged check; diff_tables_for_writes
  against tx.tables_at_begin; validation walks MvStore for
  latest_committed_begin per row; if any > begin_ts → Busy with
  rollback semantics. On success: tick clock for commit_ts; push
  committed versions; per-row apply (delete_row + restore_row) to
  db.tables; save_database for legacy WAL persistence.
- rollback_concurrent: just self.concurrent_tx.take().

New error variants:
- SQLRiteError::Busy(String) — write-write conflict.
- SQLRiteError::BusySnapshot(String) — reserved for snapshot-read
  anomalies in the 11.5 read-path integration; not emitted yet.
- SQLRiteError::is_retryable() covers both. The contract SDK
  retry helpers will rely on.

MvStore::latest_committed_begin(row_id) — returns the largest
begin_ts of any committed version on the chain, or None for
empty / in-flight-only chains. Used by commit validation.

Known limitations carried to 11.5:
- Reads via Statement::query bypass the swap (query takes &self,
  can't mutate Connection state). Reads via execute("SELECT…") DO
  see the snapshot. Full MvStore-routed reads land when the
  read-side wiring catches up.
- MVCC writes persist only via the legacy Database.tables mirror.
  An MVCC log-record WAL frame is on the 11.5 list.
- AUTOINCREMENT inside BEGIN CONCURRENT isn't specifically
  rejected; the plan flags this.
- Tables touched by concurrent writes shouldn't carry FTS / HNSW
  indexes — restore_row only maintains B-tree indexes.

Test breakdown:
- 4 ConcurrentTx struct tests in mvcc/transaction.rs.
- 11 connection-level tests covering the four plan scenarios +
  essentials (begin requires mvcc; nested rejected; legacy BEGIN
  inside concurrent rejected; DDL rejected; empty commit never
  busies; is_retryable covers Busy variants).

634/634 workspace tests pass. fmt + clippy + doc clean on
changed files. No file-format or WAL-format change.

Docs: roadmap.md marks 11.3 ✅ and 11.4 🚧 with the limitations
called out. design-decisions.md gets a 12d entry on the
deep-clone snapshot + diff-against-BEGIN trade-offs.
supported-sql.md gets a `BEGIN CONCURRENT` reference with the
retry pattern. embedding.md gets a worked transfer() example.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: joaoh82

docs/_index.md

@@ -54,7 +54,7 @@ As of May 2026, SQLRite has:
 - Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
 - SQL surface + DX follow-ups (Phase 9 complete, v0.2.0 → v0.9.1): DDL completeness — `DEFAULT`, `DROP TABLE` / `DROP INDEX`, `ALTER TABLE` (9a); free-list + manual `VACUUM` (9b) + auto-VACUUM (9c); `IS NULL` / `IS NOT NULL` (9d); `GROUP BY` + aggregates + `DISTINCT` + `LIKE` + `IN` (9e); four flavors of `JOIN` — INNER, LEFT, RIGHT, FULL OUTER (9f); prepared statements + `?` parameter binding with a per-connection LRU plan cache (9g); HNSW probe widened to cosine + dot via `WITH (metric = …)` (9h); `PRAGMA` dispatcher with the `auto_vacuum` knob (9i)
 - Benchmarks against SQLite + DuckDB (Phase 10 complete, SQLR-4 / SQLR-16): twelve-workload bench harness with a pluggable `Driver` trait, criterion-driven, pinned-host runs published. See [`docs/benchmarks.md`](benchmarks.md).
-- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 + 11.2: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock` and `ActiveTxRegistry`. WAL header v1 → v2 persists the clock high-water mark. **11.3 `MvStore` + `PRAGMA journal_mode`: shipped on this branch.** New `MvStore` (the in-memory version index keyed by `RowID`, with the snapshot-isolation visibility rule `begin <= T < end`) and the `JournalMode { Wal, Mvcc }` per-database toggle reachable via `PRAGMA journal_mode = mvcc;`. The executor doesn't consult `MvStore` yet — that wiring lands in 11.4 alongside `BEGIN CONCURRENT` writes (read-side and write-side are coupled). Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
+- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.3: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, and `MvStore` (the in-memory version index, snapshot-isolation visibility rule `begin <= T < end`). WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. **11.4 `BEGIN CONCURRENT` writes + commit-time validation: shipped on this branch.** Multiple sibling `Connection`s can each hold their own open `BEGIN CONCURRENT`; commits validate against `MvStore` and abort with `SQLRiteError::Busy` on row-level write-write conflict. The four plan-required tests pass: disjoint inserts both commit, same-row updates collide and one wins, aborted writes never become visible, retry-after-`Busy` succeeds. Reads inside the transaction via `execute("SELECT…")` see the BEGIN-time snapshot through a swap-based dispatch; reads via `Statement::query` still go through the legacy path (full `MvStore`-routed reads land in 11.5). Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
 - A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)
 
 See the [Roadmap](roadmap.md) for the full phase plan.

docs/design-decisions.md

@@ -190,6 +190,31 @@ Decisions are grouped by the engine layer they concern: parser, storage, concurr
 
 ---
 
+### 12d. `BEGIN CONCURRENT` reuses the legacy deep-clone snapshot mechanism (Phase 11.4)
+
+**Decision.** A `BEGIN CONCURRENT` transaction allocates a per-`Connection` [`ConcurrentTx`](../src/mvcc/transaction.rs) carrying:
+
+- a `TxHandle` (RAII registry entry from Phase 11.2),
+- `tables: HashMap<String, Table>` — a **deep clone** of `Database::tables` taken at BEGIN, mutated by the executor through every statement of the transaction,
+- `tables_at_begin: HashMap<String, Table>` — an **immutable second deep clone** of the same, untouched for the transaction's lifetime,
+- `schema_at_begin: Vec<String>` — sorted table-name fingerprint at BEGIN.
+
+Each statement inside the transaction runs against the working `tables` clone via a swap-and-restore (`std::mem::swap(db.tables, tx.tables)` → run executor → swap back); the executor itself doesn't know it's running inside an MVCC transaction. At COMMIT, the write-set is derived by diffing `tables_at_begin` against `tables`. Validation walks `MvStore` for the latest committed `begin_ts` per touched row; if any exceeds `tx.begin_ts` we abort with [`SQLRiteError::Busy`](../src/error.rs). On success we tick the clock for `commit_ts`, push each write into `MvStore`, apply the writes per-row to `db.tables`, and persist via the legacy `save_database`.
+
+**Why deep clones rather than a tx-local write-set + read-through-overlay.** The tx-local-overlay model (every executor read consults a tx-local map first, then the live database) is the textbook "right" answer and what 11.5+ will eventually adopt once reads route through `MvStore`. For 11.4 we wanted to ship the four plan-required tests — disjoint inserts both commit, same-row updates collide with `Busy`, aborted writes invisible, retry succeeds — without rewriting the executor. The swap-and-restore approach gets us there because the executor's `&mut Database` signature stays unchanged: from inside the swap, `db.tables` IS the snapshot. Statements inside the transaction therefore see their own writes correctly without any executor-level changes.
+
+**Why the second clone (`tables_at_begin`).** Without it, the COMMIT-time diff would be against the *current* `db.tables`, which might already carry commits from other concurrent transactions that landed between our BEGIN and our COMMIT. Their disjoint writes would surface in our diff as bogus DELETEs, and per-row apply would silently undo someone else's commit. Diffing against the BEGIN-time snapshot keeps our write-set scoped to changes we actually made. The doubled per-transaction memory is the v0 cost of correctness; column-level COW or `Arc<Table>` sharing is an obvious follow-up.
+
+**Why reads via `Statement::query` don't see the swap.** `Statement::query` takes `&self`, not `&mut self`, so it can't perform the swap (the swap mutates state on the `Connection`). Reads via `Connection::execute("SELECT …")` (which takes `&mut self`) work, because they go through `execute_in_concurrent_tx` and the swap. Phase 11.5 routes reads through `MvStore` directly — the data structure already implements the snapshot-isolation visibility rule; only the executor wiring is missing — at which point the swap path can be retired.
+
+**Why per-connection rather than per-database.** Each open `BEGIN CONCURRENT` needs its own snapshot. Putting the snapshot on `Database` would limit us to one open concurrent transaction at a time, defeating the headline concurrency story. Per-`Connection` state means N sibling `Connection::connect()` handles can each hold their own open transaction — and the database mutex still serializes per-statement execution, so the storage layer's invariants don't change.
+
+**Why DDL is rejected inside `BEGIN CONCURRENT`.** Schema mutations interact poorly with the swap-and-diff model: a CREATE TABLE inside the transaction would land on the snapshot clone but not on the live database, and the per-row apply at COMMIT can't merge a new table back. Rather than hold up 11.4 on a clean DDL story, v0 rejects with a typed error — matching the plan's explicit non-goal — and a follow-up extends the merge logic if real workloads need it.
+
+**Plan-doc reference.** [`concurrent-writes-plan.md`](concurrent-writes-plan.md) §4.5 (commit protocol), §6 (SQL surface), §8 (non-goals: DDL, AUTOINCREMENT, snapshot-isolation reads outside `BEGIN CONCURRENT`).
+
+---
+
 ## Query execution
 
 ### 13. `NULL`-as-false in `WHERE` clauses

docs/embedding.md

@@ -90,10 +90,46 @@ for h in writers { h.join().unwrap()?; }
 # Ok::<(), sqlrite::SQLRiteError>(())
 ```
 
-Today every commit still serializes through the per-database mutex (and the pager's existing process-level `flock`); the goal of 11.1 is *capability*, not throughput. True multi-writer throughput on disjoint rows arrives with `BEGIN CONCURRENT` in 11.4 — see [`concurrent-writes-plan.md`](concurrent-writes-plan.md).
+Today every commit still serializes through the per-database mutex (and the pager's existing process-level `flock`); the goal of 11.1 is *capability*, not throughput. True multi-writer throughput on disjoint rows arrives with `BEGIN CONCURRENT` in 11.4 — see below + [`concurrent-writes-plan.md`](concurrent-writes-plan.md).
 
 Per-handle state — the prepared-statement cache (LRU populated by `prepare_cached`), the cache capacity setter — stays on each handle, by design (no extra mutex traffic for a per-thread accelerator). The shared state is the `Database` (tables, pager, transaction snapshot, auto-VACUUM threshold).
 
+### Concurrent writes via `BEGIN CONCURRENT` (Phase 11.4)
+
+*Phase 11.4 — see [`supported-sql.md`](supported-sql.md#begin-concurrent-phase-114-sqlr-22) for the full SQL reference.* Multi-writer concurrency is opt-in: `PRAGMA journal_mode = mvcc;` once per database, then each writer wraps its work in `BEGIN CONCURRENT;` … `COMMIT;`. Sibling [`Connection::connect`](#sharing-one-database-across-threads) handles can each hold their own open `BEGIN CONCURRENT`; commits are validated against the [`MvStore`](../src/mvcc/store.rs) version index and abort with `SQLRiteError::Busy` if another writer superseded one of our rows.
+
+```rust
+use sqlrite::{Connection, SQLRiteError};
+
+fn transfer(primary: &mut Connection, src: i64, dst: i64, amount: i64)
+    -> Result<(), SQLRiteError>
+{
+    let mut conn = primary.connect();
+    loop {
+        conn.execute("BEGIN CONCURRENT")?;
+        conn.execute(&format!(
+            "UPDATE accounts SET balance = balance - {amount} WHERE id = {src}"
+        ))?;
+        conn.execute(&format!(
+            "UPDATE accounts SET balance = balance + {amount} WHERE id = {dst}"
+        ))?;
+        match conn.execute("COMMIT") {
+            Ok(_) => return Ok(()),
+            Err(e) if e.is_retryable() => continue, // Busy / BusySnapshot
+            Err(e) => return Err(e),
+        }
+    }
+}
+```
+
+The retryable-error branch is the headline new flow: pick a backoff policy that suits your workload (constant, exponential, jittered) and call the same closure again. `SQLRiteError::is_retryable()` covers both `Busy` and `BusySnapshot` so callers don't have to match each variant individually.
+
+**What 11.4 doesn't yet do:**
+
+- Reads via `Statement::query` inside the transaction don't see the BEGIN-time snapshot — they go through the legacy live-database read path. Reads via `Connection::execute("SELECT …")` *do* see the snapshot, via the swap-based dispatch. Full snapshot-isolated reads land in 11.5.
+- DDL inside `BEGIN CONCURRENT` is rejected with a typed error.
+- The transaction's write-set persists only via the legacy `Database::tables` mirror — a crash mid-transaction loses everything (correct behaviour, the transaction never committed). Phase 11.5 introduces an MVCC log-record WAL frame so `BEGIN CONCURRENT` writes become durable through `MvStore` itself.
+
 ### What's deferred
 
 - **Parameter binding.** `stmt.query(&[&"alice"])` is the intended shape but the current implementation takes no arguments — use string interpolation for now. Parameter binding lands with the cursor refactor.

docs/roadmap.md

@@ -599,19 +599,32 @@ Lift SQLRite past SQLite's single-writer ceiling with multi-version concurrency
 
 WAL format bumps **v1 → v2**: bytes 24..32 of the WAL header (previously reserved-zero) now carry the persisted `clock_high_water` `u64`. v1 WALs open cleanly — those zero bytes read as "clock never advanced" — and the next checkpoint rewrites the header at v2. No offline upgrade step. `Wal::set_clock_high_water` / `Wal::clock_high_water` accessors expose the field; the setter rejects regressions with a typed error.
 
-### 🚧 Phase 11.3 — `MvStore` skeleton + `PRAGMA journal_mode` opt-in *(in progress, plan-doc "Phase 10.3")*
+### ✅ Phase 11.3 — `MvStore` skeleton + `PRAGMA journal_mode` opt-in *(plan-doc "Phase 10.3")*
 
 Standalone version-index data structure + the per-database journal-mode toggle.
 
 - New [`MvStore`](../src/mvcc/store.rs): `Mutex<HashMap<RowID, Arc<RwLock<Vec<RowVersion>>>>>`. `RowID = (table, rowid)`; each `RowVersion` carries `begin: TxTimestampOrId`, `end: Option<TxTimestampOrId>`, `payload: VersionPayload` (`Present(cols)` or `Tombstone`). `MvStore::read(row, begin_ts)` implements the textbook snapshot-isolation visibility rule (`begin <= T < end`). `push_committed` validates monotonicity + caps the previous latest version's `end`; `push_in_flight` adds a placeholder version that's invisible to other readers until commit rewrites its `begin`.
 - New [`JournalMode`](../src/mvcc/mod.rs) enum (`Wal` default, `Mvcc`); per-database setting on `Database`. `PRAGMA journal_mode = wal | mvcc;` toggles; `PRAGMA journal_mode;` returns the current value as a single-row, single-column result. `Connection::journal_mode()` reads the value through the public API. Switching `Mvcc → Wal` is rejected if the store carries committed versions (would silently strand them); v0 is intentionally strict.
 - `Database` grows `mvcc_clock: Arc<MvccClock>` and `mv_store: MvStore` fields, allocated on every `Database::new` so the toggle to MVCC mode doesn't require a re-init step. Both are shared across every `Connection::connect` sibling.
 
-The executor doesn't consult `MvStore` yet — that wiring lives in 11.4 alongside `BEGIN CONCURRENT` writes (the read-side and write-side are coupled: snapshot reads make sense only once the commit path is mirroring versions into the store). 11.3's contract is *the data structure + the toggle exist and round-trip*; 11.4 will turn the dial.
+### 🚧 Phase 11.4 — `BEGIN CONCURRENT` writes + commit-time validation *(in progress, plan-doc "Phase 10.4" — the meat)*
 
-### Phase 11.4 — `BEGIN CONCURRENT` writes + commit-time validation *(planned, the meat)*
+The headline slice. Multiple sibling `Connection`s can each hold their own open `BEGIN CONCURRENT` transaction; commits validate against `MvStore` and abort with [`SQLRiteError::Busy`](../src/error.rs) on row-level write-write conflict. The four plan-required tests pass: disjoint inserts both commit, same-row updates collide and one wins, aborted writes never become visible, retry-after-`Busy` succeeds.
 
-Parser maps `BEGIN CONCURRENT` to `TxKind::Concurrent`; writes land in the MvStore's write-set; commit walks the write-set checking for newer versions. New `SQLRiteError::Busy` / `BusySnapshot` variants. New WAL log-record frame kind.
+- New [`ConcurrentTx`](../src/mvcc/transaction.rs) — per-`Connection` state holding the [`TxHandle`](../src/mvcc/registry.rs) (RAII registry entry, drops at COMMIT/ROLLBACK), a private deep-clone of `Database::tables` (working state — what each statement's executor mutates), and an immutable second clone (`tables_at_begin` — used at COMMIT to derive the write-set without seeing other transactions' commits). The doubled per-tx memory is the v0 trade for correctness; column-level COW ↔ shared-`Arc` table cloning is the obvious follow-up.
+- `Connection` grows a `concurrent_tx: Option<ConcurrentTx>` field, plus three new methods: `begin_concurrent()`, `commit_concurrent()`, `rollback_concurrent()`. `Connection::execute` intercepts `BEGIN CONCURRENT` / `COMMIT` / `ROLLBACK` before sqlparser runs (sqlparser 0.61 doesn't have a `Concurrent` modifier — same intercept pattern as PRAGMA).
+- Inside an open concurrent transaction, every other statement runs against the transaction's private cloned tables: `Connection::execute_in_concurrent_tx` swaps `db.tables` ↔ `tx.tables` for the duration of the executor call, parks a dummy `TxnSnapshot` on `db.txn` to suppress the auto-save, runs `process_command`, then unwinds in reverse. The executor itself doesn't change.
+- COMMIT shape (Hekaton-style optimistic validation): diff `tx.tables_at_begin` vs `tx.tables` → write-set; for each row, walk `MvStore` for the latest committed version's `begin_ts` (new `MvStore::latest_committed_begin` accessor); abort with `Busy` if any latest exceeds `tx.begin_ts`. On success: tick the clock for `commit_ts`, push every write into `MvStore` as a committed version (auto-caps the previous latest's `end`), apply per-row to `db.tables` (`delete_row` then `restore_row` — preserves secondary B-tree indexes), and run the legacy `save_database` so changes persist via the existing WAL.
+- ROLLBACK is just `self.concurrent_tx.take()` — the cloned tables drop, the `TxHandle` unregisters, the live database was never touched.
+- New `SQLRiteError::Busy(String)` and `SQLRiteError::BusySnapshot(String)` variants. `SQLRiteError::is_retryable()` covers both — the contract SDK retry helpers will rely on.
+- DDL inside `BEGIN CONCURRENT` (CREATE TABLE / CREATE INDEX / DROP TABLE / DROP INDEX / ALTER TABLE / VACUUM) is rejected before the swap with a typed error (plan §8 non-goal).
+
+**Known limitations carried into 11.5+:**
+
+- Reads via `Statement::query` / `Statement::query_with_params` (the prepared-statement path) bypass the swap and read from the live `Database::tables`. Reads via `Connection::execute("SELECT…")` go through the swap and see the snapshot. Full snapshot-isolation reads land when 11.5 routes reads through `MvStore` — the data structure already implements `begin <= T < end` visibility; only the executor wiring is missing.
+- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.5 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror.
+- `AUTOINCREMENT` inside `BEGIN CONCURRENT` isn't explicitly rejected; the v0 deep-clone-snapshot model handles concurrent INSERTs by isolating each tx's `last_rowid` bumps to its private snapshot, so two concurrent INSERTs on an `AUTOINCREMENT` column may collide at COMMIT and surface as `Busy`. Adopting the plan's "reject AUTOINCREMENT under MVCC" gate is a clean follow-up.
+- Tables touched by `BEGIN CONCURRENT` writes can't carry FTS or HNSW indexes today — `restore_row` only maintains B-tree secondary indexes. Concurrent-tx tests don't exercise FTS / HNSW, but a runtime guard would surface this with a clear error rather than producing inconsistent indexes.
 
 ### Phase 11.5 — Checkpoint integration + crash recovery *(planned)*