Skip to content

Commit e84801e

Browse files
joaoh82claude
andauthored
feat(sdk): Phase 11.8 multi-handle SDK shape (SQLR-22) (#129)
Closes the end-to-end gap from 11.7. The retry-error machinery (BusyError / errorKind / ErrBusy) was reachable through every SDK after 11.7 — but each `sqlrite.connect()` / `new Database()` / `sql.Open()` built an *isolated* backing DB, so the retry idioms were untriggerable. This slice exposes the engine's `Connection::connect()` through the public SDK surfaces so apps can finally mint sibling handles that share state. Picked ahead of plan-doc 11.5 (durability) for the same reason 11.5–11.7 jumped the queue: visible user value first. After this PR, the canonical `BEGIN CONCURRENT` retry loop pattern works end-to-end through Python and Node. C FFI ([`sqlrite-ffi/src/lib.rs`]): - new `sqlrite_connect_sibling(existing, out)` function. Thin wrapper around the engine's `Connection::connect`. Sibling has its own pointer + lifecycle but shares Database state. - header regenerated via build.rs. - 2 new tests: connect_sibling_mints_a_handle_that_shares_state (real cross-handle BEGIN CONCURRENT conflict through the public API) and connect_sibling_rejects_null_inputs. Python SDK ([`sdk/python/src/lib.rs`]): - new `Connection.connect()` instance method. Acquires the inner Mutex<RustConnection>, calls `inner.connect()`, wraps result in a fresh pyclass with its own Mutex. Inherits the parent's ask_config. - 4 new tests: sibling sharing, outliving closed parent, raise on closed connect, and the headline busy-round-trip-via-siblings. Node.js SDK ([`sdk/nodejs/src/lib.rs`]): - new `Database.connect()` method on the napi-rs class. Same shape — sibling Database with its own RefCell + ask_config clone. - 4 new tests including the cross-sibling BusyError round trip using 11.7's errorKind classifier. Go SDK: deliberately skipped. database/sql's pool model already gives sibling-like behavior across `db.Conn(ctx)` calls within a single `sql.Open`, but exposing a *cross-pool* sibling shape through `database/sql` would need a process-level path → Database registry. Deferred to a follow-up (roadmap 11.11). WASM SDK: deliberately skipped (single-threaded browser + wasm-bindgen lifetime complications). Same deferral as 11.7. Workspace: 654/654 Rust tests pass (was 652 + 2 new FFI). Python + Node SDK CI jobs will exercise the new sibling-handle tests on the next CI run. fmt + clippy + doc clean on changed files. Roadmap renumbered again: - plan-doc 11.5 (checkpoint) → roadmap 11.9 (was 11.8) - plan-doc 11.7 (indexes) → roadmap 11.10 (was 11.9) - REPL .spawn + bench workload + Go multi-handle = roadmap 11.11 - plan-doc 11.9 (docs) → roadmap 11.12 (was 11.11) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 358d995 commit e84801e

8 files changed

Lines changed: 402 additions & 18 deletions

File tree

docs/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ As of May 2026, SQLRite has:
5454
- Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
5555
- SQL surface + DX follow-ups (Phase 9 complete, v0.2.0 → v0.9.1): DDL completeness — `DEFAULT`, `DROP TABLE` / `DROP INDEX`, `ALTER TABLE` (9a); free-list + manual `VACUUM` (9b) + auto-VACUUM (9c); `IS NULL` / `IS NOT NULL` (9d); `GROUP BY` + aggregates + `DISTINCT` + `LIKE` + `IN` (9e); four flavors of `JOIN` — INNER, LEFT, RIGHT, FULL OUTER (9f); prepared statements + `?` parameter binding with a per-connection LRU plan cache (9g); HNSW probe widened to cosine + dot via `WITH (metric = …)` (9h); `PRAGMA` dispatcher with the `auto_vacuum` knob (9i)
5656
- Benchmarks against SQLite + DuckDB (Phase 10 complete, SQLR-4 / SQLR-16): twelve-workload bench harness with a pluggable `Driver` trait, criterion-driven, pinned-host runs published. See [`docs/benchmarks.md`](benchmarks.md).
57-
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.6: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, and `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes go through commit-time validation and abort with `SQLRiteError::Busy` on row-level write-write conflict. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `Connection::vacuum_mvcc()` bound the in-memory version chain growth. **11.7 SDK propagation: shipped on this branch.** The C FFI gains `SqlriteStatus::Busy` / `BusySnapshot`. Python adds `sqlrite.BusyError` / `BusySnapshotError` subclasses. Node exports `errorKind(message)` + `ErrorKind` enum. Go adds `ErrBusy` / `ErrBusySnapshot` sentinels (matchable with `errors.Is`) + an `IsRetryable(err)` helper. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
57+
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.7: shipped.** Engine + SDK error propagation: `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes commit-validate against `MvStore` and abort with `SQLRiteError::Busy`. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `vacuum_mvcc()` bound the version chain growth. C FFI / Python / Node / Go all propagate `Busy` / `BusySnapshot` as typed retryable errors. **11.8 multi-handle SDK shape: shipped on this branch.** The FFI's `sqlrite_connect_sibling`, Python's `Connection.connect()`, and Node's `db.connect()` mint sibling handles that share backing state — closes the end-to-end gap from 11.7 where `BusyError` was reachable but not exerciseable through any SDK. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
5858
- A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)
5959

6060
See the [Roadmap](roadmap.md) for the full phase plan.

docs/roadmap.md

Lines changed: 23 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -622,7 +622,7 @@ The headline slice. Multiple sibling `Connection`s can each hold their own open
622622
**Known limitations carried forward (most resolved in 11.5):**
623623

624624
- ~~Reads via `Statement::query` / `Statement::query_with_params` bypass the swap.~~ ✅ Fixed in 11.5 — `Connection.concurrent_tx` is now `Mutex<Option<…>>` and a new `with_snapshot_read` helper threads the swap through `&self`.
625-
- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.8 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.)
625+
- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.9 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.)
626626
- `AUTOINCREMENT` inside `BEGIN CONCURRENT` isn't explicitly rejected; the v0 deep-clone-snapshot model handles concurrent INSERTs by isolating each tx's `last_rowid` bumps to its private snapshot, so two concurrent INSERTs on an `AUTOINCREMENT` column may collide at COMMIT and surface as `Busy`. Adopting the plan's "reject AUTOINCREMENT under MVCC" gate is a clean follow-up.
627627
- Tables touched by `BEGIN CONCURRENT` writes can't carry FTS or HNSW indexes today — `restore_row` only maintains B-tree secondary indexes. Concurrent-tx tests don't exercise FTS / HNSW, but a runtime guard would surface this with a clear error rather than producing inconsistent indexes.
628628

@@ -647,34 +647,41 @@ Bounds in-memory growth of the [`MvStore`](../src/mvcc/store.rs) version chains.
647647
**What 11.6 doesn't yet do:**
648648

649649
- No background GC thread or `PRAGMA mvcc_gc_interval_ms`. Per-commit sweep + explicit `vacuum_mvcc()` cover the v0 model; the periodic-sweep variant lands as a follow-up if profiles show it's needed.
650-
- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.8.
650+
- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.9.
651651

652-
### 🚧 Phase 11.7 — SDK propagation of `Busy` / `BusySnapshot` *(in progress, plan-doc "Phase 10.8"; promoted ahead of plan-doc 11.5 checkpoint work for the same reason 11.5 / 11.6 jumped the queue — surfacing retryable errors to SDK callers is what unblocks Python / Node / Go users from actually writing `BEGIN CONCURRENT` retry loops)*
652+
### Phase 11.7 — SDK propagation of `Busy` / `BusySnapshot` *(plan-doc "Phase 10.8"'s first half; promoted ahead of plan-doc 11.5 checkpoint work because surfacing retryable errors to SDK callers is what unblocks Python / Node / Go users from writing `BEGIN CONCURRENT` retry loops)*
653653

654-
- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `SqlriteStatus::Busy = 5` and `SqlriteStatus::BusySnapshot = 6` codes alongside the existing `Ok` / `Error` / `InvalidArgument` set. `SqlriteStatus::is_retryable()` covers both. A new internal `status_of_sqlrite` mapper inspects the engine's `SQLRiteError` variant and routes `Busy` / `BusySnapshot` to the dedicated codes (the generic `status_of` keeps mapping every error to `Error`). `sqlrite_execute` switches to the engine-aware mapper so `BEGIN CONCURRENT` commits surface the dedicated codes through every language binding. Header regenerated automatically via `build.rs`.
655-
- **Python SDK** ([`sdk/python/src/lib.rs`](../sdk/python/src/lib.rs)): two new exception classes `sqlrite.BusyError` and `sqlrite.BusySnapshotError`, both inheriting from `sqlrite.SQLRiteError`. Existing `except sqlrite.SQLRiteError` blocks keep catching them; retry helpers can branch with `except sqlrite.BusyError`. A new `map_engine_err` helper inspects the engine error variant and raises the matching exception class. Every engine-typed call site (open / execute / prepare / query / rows.next) routes through it.
656-
- **Node.js SDK** ([`sdk/nodejs/src/lib.rs`](../sdk/nodejs/src/lib.rs)): new exported `ErrorKind` string enum (`'Busy'`, `'BusySnapshot'`, `'Other'`) and `errorKind(message: string)` classifier function. The engine's `thiserror` Display already prefixes retryable errors with `'Busy: '` / `'BusySnapshot: '`, so the classifier just regex-tests the prefix. JS callers wrap their `BEGIN CONCURRENT` loops in `try / catch (err) { if (errorKind(err.message) === ErrorKind.Busy) continue; }`.
657-
- **Go SDK** ([`sdk/go/sqlrite.go`](../sdk/go/sqlrite.go)): two new sentinel error values `sqlrite.ErrBusy` and `sqlrite.ErrBusySnapshot`, plus an `IsRetryable(err error) bool` helper. `wrapErr` recognises the new FFI status codes and wraps the engine message with `fmt.Errorf("…: %w", ErrBusy)` so `errors.Is(err, sqlrite.ErrBusy)` works through the `database/sql` driver chain.
658-
- **WASM SDK** — deliberately untouched. The browser WASM target is single-threaded; `BEGIN CONCURRENT` is meaningful but multi-handle concurrency through `Connection::connect` isn't yet exposed across `wasm-bindgen`'s lifetime model. When the multi-handle JS shape lands (separate slice), the same `Busy: …` message prefix will be the classifier hook for the WASM bindings too.
654+
- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `SqlriteStatus::Busy = 5` and `SqlriteStatus::BusySnapshot = 6` codes; `SqlriteStatus::is_retryable()` covers both. A new internal `status_of_sqlrite` mapper inspects the engine's `SQLRiteError` variant and routes `Busy` / `BusySnapshot` to the dedicated codes.
655+
- **Python SDK**: two new exception classes `sqlrite.BusyError` and `sqlrite.BusySnapshotError`, both inheriting from `sqlrite.SQLRiteError`. `map_engine_err` helper raises the matching subclass.
656+
- **Node.js SDK**: exported `ErrorKind` string enum (`'Busy'`, `'BusySnapshot'`, `'Other'`) and `errorKind(message: string)` classifier function. The engine's `thiserror` Display prefixes retryable errors with `'Busy: '` / `'BusySnapshot: '` so the classifier just regex-tests the prefix.
657+
- **Go SDK**: two new sentinel error values `sqlrite.ErrBusy` / `sqlrite.ErrBusySnapshot`, plus an `IsRetryable(err error) bool` helper. `wrapErr` recognises the new FFI status codes and wraps the engine message with `fmt.Errorf("…: %w", ErrBusy)`.
658+
- **WASM SDK** — deliberately untouched (browser is single-threaded; multi-handle shape not yet exposed).
659659

660-
**What this slice doesn't do:**
660+
### 🚧 Phase 11.8 — Multi-handle SDK shape *(in progress, was plan-doc 11.8's other half; promoted ahead of plan-doc 11.5 again because the 11.7 retry-error machinery can't be exercised end-to-end through any SDK until siblings are reachable)*
661661

662-
- The multi-handle / sibling-`Connection` shape isn't exposed through any SDK yet. Each `sqlrite.connect(path)` / `new Database(path)` / `sql.Open(...)` builds an independent backing database. End-to-end testing of cross-handle `Busy` is therefore deferred to the multi-handle SDK slice; this PR ships the *plumbing* so once that wiring lands, callers already have the retry idioms they need.
663-
- The WAL log-record durability work (plan-doc 11.5 / our 11.8) stays deferred.
662+
Each pre-11.8 SDK `connect()` / `new Database()` built an *isolated* backing DB; the 11.7 `BusyError` / `errorKind` / `ErrBusy` plumbing was reachable but not actually triggerable from user code. This slice exposes the engine's `Connection::connect()` through every reachable language so apps can mint sibling handles that share state, and finally exercise the 11.7 retry idioms with real cross-handle conflicts.
664663

665-
### Phase 11.8 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow GC + SDK propagation because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)*
664+
- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `sqlrite_connect_sibling(existing, out)` function. Wraps the engine's `Connection::connect`. Callers get a sibling handle with its own `SqlriteConnection` pointer but shared backing database; the sibling must be closed via `sqlrite_close` (its lifecycle is independent — closing one handle doesn't tear down the others while a sibling is still alive).
665+
- **Python SDK** ([`sdk/python/src/lib.rs`](../sdk/python/src/lib.rs)): new `Connection.connect()` instance method that mints a sibling pyclass. Wraps the engine's `Connection::connect` inside the existing `Mutex<RustConnection>`. The new handle inherits the parent's `ask_config`.
666+
- **Node.js SDK** ([`sdk/nodejs/src/lib.rs`](../sdk/nodejs/src/lib.rs)): new `db.connect()` method on the `Database` class. Same shape — sibling shares state, can hold its own `BEGIN CONCURRENT`.
667+
- **Go SDK** — deliberately not changed. Go's `database/sql` already gives callers a connection pool over a single `sql.Open`; each pool connection acquired through `db.Conn(ctx)` is *already* a sibling of the rest at the driver layer. But each `sql.Open("sqlrite", path)` still builds an independent backing DB because the pool is per-`sql.DB`. Exposing a cross-pool sibling shape through the `database/sql` driver model is genuinely non-obvious (it'd require a process-level registry keyed by path); deferred to the multi-handle Go follow-up.
668+
- **WASM SDK** — still untouched. The browser is single-threaded and `wasm-bindgen` lifetimes complicate sibling pyclass-style sharing. Same deferral as 11.7.
669+
670+
Each SDK gets end-to-end tests that exercise `BEGIN CONCURRENT` cross-handle conflicts: two sibling handles, two concurrent transactions on the same row, the second commit hits the SDK's typed retryable error, retry succeeds.
671+
672+
### Phase 11.9 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow SDK propagation because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)*
666673

667674
MVCC log-record WAL frame format (the deferred 11.4 piece). Commit appends log records pre-`save_database`. Reopen replays log records into `MvStore`. Checkpoint drains `MvStore` versions back into the pager (so `Mvcc → Wal` becomes legal once the store is empty). Crash-recovery test: kill mid-commit between log-record append and version-chain push; reopen; verify the committed transaction is visible and the half-written one is not.
668675

669-
### Phase 11.9 — Indexes under MVCC *(deferred-by-design, plan-doc "Phase 10.7")*
676+
### Phase 11.10 — Indexes under MVCC *(deferred-by-design, plan-doc "Phase 10.7")*
670677

671678
Each secondary-index entry becomes its own `RowVersion`. Turso explicitly punted on this; SQLRite's v0 will reject `CREATE INDEX` while `journal_mode = mvcc`.
672679

673-
### Phase 11.10Multi-handle SDK shape + REPL `.spawn` *(planned, was plan-doc 11.8's other half)*
680+
### Phase 11.11 — REPL `.spawn` + bench workload *(planned)*
674681

675-
Expose `Connection::connect()` through the FFI + each SDK so Python / Node / Go callers can mint sibling handles, plus a new REPL `.spawn` meta-command. Without this, the 11.7 retry-error machinery can't actually be exercised end-to-end through an SDK (each SDK `connect()` builds an independent DB). Also adds the "N concurrent writers" benchmark workload.
682+
REPL `.spawn` meta-command for interactive `BEGIN CONCURRENT` demos. New "N concurrent writers" benchmark workload pitting SQLRite-MVCC against SQLite + DuckDB on disjoint-row write throughput. Plus Go SDK multi-handle work (cross-pool sibling shape).
676683

677-
### Phase 11.11 — Docs *(planned, plan-doc "Phase 10.9")*
684+
### Phase 11.12 — Docs *(planned, plan-doc "Phase 10.9")*
678685

679686
Promote the plan to `docs/concurrent-writes.md` and update the cross-references.
680687

sdk/nodejs/src/lib.rs

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,6 +299,40 @@ impl Database {
299299
})
300300
}
301301

302+
/// Phase 11.8 — mints a sibling `Database` that shares the
303+
/// same underlying state (the in-memory tables, the MVCC
304+
/// store, the pager). Wraps the engine's `Connection::connect`.
305+
///
306+
/// Use this to drive `BEGIN CONCURRENT` from multiple Node
307+
/// handles in the same process: each sibling can hold its own
308+
/// concurrent transaction, and commits validate against the
309+
/// shared MvStore.
310+
///
311+
/// ```js
312+
/// const db = new Database(':memory:');
313+
/// db.exec('PRAGMA journal_mode = mvcc');
314+
/// db.exec('CREATE TABLE t (id INTEGER PRIMARY KEY, v INTEGER)');
315+
/// const sibling = db.connect();
316+
/// // sibling.exec(...) lands on the same backing tables.
317+
/// ```
318+
///
319+
/// The sibling carries its own per-handle prepared-statement
320+
/// cache and concurrent-transaction slot. Closing one handle
321+
/// (`db.close()`) doesn't affect siblings; the underlying
322+
/// database lives as long as any handle holds it.
323+
#[napi]
324+
pub fn connect(&self) -> Result<Database> {
325+
let borrow = self.inner.borrow();
326+
let parent = borrow
327+
.as_ref()
328+
.ok_or_else(|| napi::Error::from_reason("cannot connect: database is closed"))?;
329+
let sibling = parent.connect();
330+
Ok(Database {
331+
inner: RefCell::new(Some(sibling)),
332+
ask_config: RefCell::new(self.ask_config.borrow().clone()),
333+
})
334+
}
335+
302336
/// Closes the connection and releases the OS file lock. Safe to
303337
/// call multiple times.
304338
#[napi]

0 commit comments

Comments
 (0)