You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* feat(sdk): Phase 11.7 SDK propagation of Busy / BusySnapshot (SQLR-22)
Surfaces retryable engine errors through the C FFI and every
language SDK so Python / Node / Go callers can actually write
BEGIN CONCURRENT retry loops with idiomatic per-language
patterns. Picked ahead of plan-doc 11.5 (checkpoint integration)
for the same reason 11.5 / 11.6 jumped the queue — durability
already works through the legacy save_database mirror, but
SDK users hitting BEGIN CONCURRENT had no way to distinguish
retryable errors from real failures.
Plan-doc 11.8 ("SDK + REPL propagation") split into two:
- FFI/SDK error propagation ships here as roadmap 11.7
- Multi-handle SDK shape + REPL .spawn → roadmap 11.10
C FFI:
- new SqlriteStatus::Busy = 5 + BusySnapshot = 6 codes
- SqlriteStatus::is_retryable() covers both
- new status_of_sqlrite() mapper routes engine-typed errors to
the dedicated codes; generic status_of() keeps mapping every
error to Error for non-engine result types
- sqlrite_execute switched to the engine-aware mapper
- header regenerated via build.rs
Python SDK:
- new sqlrite.BusyError + sqlrite.BusySnapshotError pyo3
exception classes, both inheriting from sqlrite.SQLRiteError
- new map_engine_err() helper inspects the engine variant and
raises the matching exception class
- all engine-typed call sites (open / execute / prepare /
query / rows.next) routed through it
- existing `except sqlrite.SQLRiteError` blocks still catch
both; retry helpers branch with `except sqlrite.BusyError`
Node.js SDK:
- new exported ErrorKind string enum ('Busy' | 'BusySnapshot'
| 'Other') and errorKind(message) classifier function
- engine's thiserror Display already prefixes the error
message with 'Busy: ' / 'BusySnapshot: '; classifier matches
the prefix (longest-first to avoid mis-classifying snapshot
errors)
- JS pattern: try { ... } catch (err) {
if (errorKind(err.message) === ErrorKind.Busy) continue;
throw err;
}
Go SDK:
- new ErrBusy + ErrBusySnapshot sentinel errors
- new IsRetryable(err error) bool helper covers both
- wrapErr recognises new FFI status codes and wraps the
engine message with fmt.Errorf("…: %w", ErrBusy) so
errors.Is(err, sqlrite.ErrBusy) works through the cgo +
database/sql driver chain
WASM SDK: deliberately untouched. Browser is single-threaded;
multi-handle Connection::connect concurrency isn't exposed
through wasm-bindgen yet. Same 'Busy: ' message prefix will
be the classifier hook when the multi-handle WASM work lands
(11.10).
Subtle issues hit:
- napi-rs derive #[napi(string_enum)] auto-adds Clone+Copy.
My manual `#[derive(Debug, Clone, Copy, ...)]` produced
E0119 (conflicting impls). Dropped Clone+Copy from the
manual derive list.
- Python's `SQLRiteError` short name clashes between the
engine enum and the pyo3 exception class (both called
SQLRiteError). Used fully-qualified
`sqlrite::SQLRiteError::Busy(_)` in match arms.
Tests:
- 2 new FFI tests: is_retryable_covers_busy_variants +
begin_concurrent_busy_status_round_trip (exercises a real
conflict over a shared backing DB via Connection::connect
through FFI).
- 3 Node-side Rust tests for classify_error_message.
- 2 Python tests (class existence + inheritance, journal_mode
round-trip).
- 3 Go tests (sentinel distinctness, IsRetryable coverage,
journal_mode round-trip).
What this slice doesn't do:
- Each SDK's connect() still builds an independent backing DB.
End-to-end testing of cross-handle Busy needs the multi-handle
SDK shape (planned as 11.10). This PR ships the *plumbing* so
once that wiring lands, callers already have the retry idioms.
- WAL log-record durability still deferred to 11.8.
- SDK READMEs not yet updated with retry examples — that's a
doc sweep, lands alongside 11.11.
652/652 Rust workspace tests pass. 3/3 Node-side classifier
tests pass. fmt + clippy + doc clean on changed files. Python
+ Go SDK CI jobs will exercise the SDK-side additions on the
next run.
Roadmap renumbered: plan-doc 11.5 (checkpoint) → roadmap 11.8;
plan-doc 11.7 (indexes) → 11.9; multi-handle SDK + REPL =
11.10; plan-doc 11.9 (docs) → 11.11. Called out in the
roadmap entries so plan-doc references remain readable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sdk): Python + Go test fixes for CI
Go: my Phase 11.7 tests referenced the new sentinels and helper as
bare names (`ErrBusy`, `IsRetryable`), but `sdk/go/sqlrite_test.go`
is `package sqlrite_test` (external) and needs the `sqlrite.`
prefix. CI's `go test ./...` caught it; my workspace cargo build
didn't (Go SDK isn't a Rust workspace member).
Python: my `test_journal_mode_pragma_round_trips_through_python`
asserted `cur.fetchone() == ('mvcc',)` after `PRAGMA journal_mode`,
but PRAGMA goes through the cursor's non-query path — the rendered
single-row result lives in `CommandOutput.rendered`, never surfaced
through the cursor's row iterator. Fetchone() returns None.
Reshaped both tests to verify the *gate* opens rather than the
read-form renders: `PRAGMA journal_mode = mvcc` followed by a
`BEGIN CONCURRENT` that would otherwise reject. Same property,
cleanly testable through each SDK's public API today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/_index.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -54,7 +54,7 @@ As of May 2026, SQLRite has:
54
54
- Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
55
55
- SQL surface + DX follow-ups (Phase 9 complete, v0.2.0 → v0.9.1): DDL completeness — `DEFAULT`, `DROP TABLE` / `DROP INDEX`, `ALTER TABLE` (9a); free-list + manual `VACUUM` (9b) + auto-VACUUM (9c); `IS NULL` / `IS NOT NULL` (9d); `GROUP BY` + aggregates + `DISTINCT` + `LIKE` + `IN` (9e); four flavors of `JOIN` — INNER, LEFT, RIGHT, FULL OUTER (9f); prepared statements + `?` parameter binding with a per-connection LRU plan cache (9g); HNSW probe widened to cosine + dot via `WITH (metric = …)` (9h); `PRAGMA` dispatcher with the `auto_vacuum` knob (9i)
56
56
- Benchmarks against SQLite + DuckDB (Phase 10 complete, SQLR-4 / SQLR-16): twelve-workload bench harness with a pluggable `Driver` trait, criterion-driven, pinned-host runs published. See [`docs/benchmarks.md`](benchmarks.md).
57
-
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.5: shipped.**`Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, and `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes go through commit-time validation against `MvStore` and abort with `SQLRiteError::Busy` on row-level write-write conflict. Reads via `Statement::query` see the BEGIN-time snapshot. **11.6 garbage collection: shipped on this branch.** Per-commit GC sweep on the write-set's chains + a new `Connection::vacuum_mvcc()` for explicit full-store drains; bounds the in-memory version chain growth that 11.4–11.5 left unbounded. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
57
+
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.6: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, and `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes go through commit-time validation and abort with `SQLRiteError::Busy` on row-level write-write conflict. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `Connection::vacuum_mvcc()` bound the in-memory version chain growth. **11.7 SDK propagation: shipped on this branch.** The C FFI gains `SqlriteStatus::Busy` / `BusySnapshot`. Python adds `sqlrite.BusyError` / `BusySnapshotError` subclasses. Node exports `errorKind(message)` + `ErrorKind` enum. Go adds `ErrBusy` / `ErrBusySnapshot` sentinels (matchable with `errors.Is`) + an `IsRetryable(err)` helper. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
58
58
- A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)
59
59
60
60
See the [Roadmap](roadmap.md) for the full phase plan.
Copy file name to clipboardExpand all lines: docs/roadmap.md
+21-8Lines changed: 21 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -622,7 +622,7 @@ The headline slice. Multiple sibling `Connection`s can each hold their own open
622
622
**Known limitations carried forward (most resolved in 11.5):**
623
623
624
624
-~~Reads via `Statement::query` / `Statement::query_with_params` bypass the swap.~~ ✅ Fixed in 11.5 — `Connection.concurrent_tx` is now `Mutex<Option<…>>` and a new `with_snapshot_read` helper threads the swap through `&self`.
625
-
- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.7 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.)
625
+
- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.8 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.)
626
626
-`AUTOINCREMENT` inside `BEGIN CONCURRENT` isn't explicitly rejected; the v0 deep-clone-snapshot model handles concurrent INSERTs by isolating each tx's `last_rowid` bumps to its private snapshot, so two concurrent INSERTs on an `AUTOINCREMENT` column may collide at COMMIT and surface as `Busy`. Adopting the plan's "reject AUTOINCREMENT under MVCC" gate is a clean follow-up.
627
627
- Tables touched by `BEGIN CONCURRENT` writes can't carry FTS or HNSW indexes today — `restore_row` only maintains B-tree secondary indexes. Concurrent-tx tests don't exercise FTS / HNSW, but a runtime guard would surface this with a clear error rather than producing inconsistent indexes.
628
628
@@ -634,7 +634,7 @@ Lock order is consistently `concurrent_tx → inner` across every code path; dea
634
634
635
635
This was renumbered out of plan-doc order: the plan-doc had 11.5 as checkpoint integration, but that's a much larger slice and the prepare/query-bypass-the-swap gap was a real correctness hole for users hitting `BEGIN CONCURRENT`. Plan-doc 11.5 (checkpoint) → roadmap 11.7; plan-doc 11.6 (GC) → roadmap 11.6 (this one).
636
636
637
-
### 🚧 Phase 11.6 — Garbage collection *(in progress, plan-doc "Phase 10.6"; promoted ahead of plan-doc 11.5 because unbounded `MvStore` growth was the next concrete user-impact concern after 11.5 closed the snapshot-read gap)*
637
+
### ✅ Phase 11.6 — Garbage collection *(plan-doc "Phase 10.6"; promoted ahead of plan-doc 11.5 because unbounded `MvStore` growth was the next concrete user-impact concern after 11.5 closed the snapshot-read gap)*
638
638
639
639
Bounds in-memory growth of the [`MvStore`](../src/mvcc/store.rs) version chains. Without this, every committed version stays forever in the in-memory chain — a memory leak that grows linearly with commits.
640
640
@@ -647,21 +647,34 @@ Bounds in-memory growth of the [`MvStore`](../src/mvcc/store.rs) version chains.
647
647
**What 11.6 doesn't yet do:**
648
648
649
649
- No background GC thread or `PRAGMA mvcc_gc_interval_ms`. Per-commit sweep + explicit `vacuum_mvcc()` cover the v0 model; the periodic-sweep variant lands as a follow-up if profiles show it's needed.
650
-
- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.7.
650
+
- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.8.
651
651
652
-
### Phase 11.7 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow GC because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)*
652
+
### 🚧 Phase 11.7 — SDK propagation of `Busy` / `BusySnapshot`*(in progress, plan-doc "Phase 10.8"; promoted ahead of plan-doc 11.5 checkpoint work for the same reason 11.5 / 11.6 jumped the queue — surfacing retryable errors to SDK callers is what unblocks Python / Node / Go users from actually writing `BEGIN CONCURRENT` retry loops)*
653
+
654
+
-**C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `SqlriteStatus::Busy = 5` and `SqlriteStatus::BusySnapshot = 6` codes alongside the existing `Ok` / `Error` / `InvalidArgument` set. `SqlriteStatus::is_retryable()` covers both. A new internal `status_of_sqlrite` mapper inspects the engine's `SQLRiteError` variant and routes `Busy` / `BusySnapshot` to the dedicated codes (the generic `status_of` keeps mapping every error to `Error`). `sqlrite_execute` switches to the engine-aware mapper so `BEGIN CONCURRENT` commits surface the dedicated codes through every language binding. Header regenerated automatically via `build.rs`.
655
+
-**Python SDK** ([`sdk/python/src/lib.rs`](../sdk/python/src/lib.rs)): two new exception classes `sqlrite.BusyError` and `sqlrite.BusySnapshotError`, both inheriting from `sqlrite.SQLRiteError`. Existing `except sqlrite.SQLRiteError` blocks keep catching them; retry helpers can branch with `except sqlrite.BusyError`. A new `map_engine_err` helper inspects the engine error variant and raises the matching exception class. Every engine-typed call site (open / execute / prepare / query / rows.next) routes through it.
656
+
-**Node.js SDK** ([`sdk/nodejs/src/lib.rs`](../sdk/nodejs/src/lib.rs)): new exported `ErrorKind` string enum (`'Busy'`, `'BusySnapshot'`, `'Other'`) and `errorKind(message: string)` classifier function. The engine's `thiserror` Display already prefixes retryable errors with `'Busy: '` / `'BusySnapshot: '`, so the classifier just regex-tests the prefix. JS callers wrap their `BEGIN CONCURRENT` loops in `try / catch (err) { if (errorKind(err.message) === ErrorKind.Busy) continue; }`.
657
+
-**Go SDK** ([`sdk/go/sqlrite.go`](../sdk/go/sqlrite.go)): two new sentinel error values `sqlrite.ErrBusy` and `sqlrite.ErrBusySnapshot`, plus an `IsRetryable(err error) bool` helper. `wrapErr` recognises the new FFI status codes and wraps the engine message with `fmt.Errorf("…: %w", ErrBusy)` so `errors.Is(err, sqlrite.ErrBusy)` works through the `database/sql` driver chain.
658
+
-**WASM SDK** — deliberately untouched. The browser WASM target is single-threaded; `BEGIN CONCURRENT` is meaningful but multi-handle concurrency through `Connection::connect` isn't yet exposed across `wasm-bindgen`'s lifetime model. When the multi-handle JS shape lands (separate slice), the same `Busy: …` message prefix will be the classifier hook for the WASM bindings too.
659
+
660
+
**What this slice doesn't do:**
661
+
662
+
- The multi-handle / sibling-`Connection` shape isn't exposed through any SDK yet. Each `sqlrite.connect(path)` / `new Database(path)` / `sql.Open(...)` builds an independent backing database. End-to-end testing of cross-handle `Busy` is therefore deferred to the multi-handle SDK slice; this PR ships the *plumbing* so once that wiring lands, callers already have the retry idioms they need.
663
+
- The WAL log-record durability work (plan-doc 11.5 / our 11.8) stays deferred.
664
+
665
+
### Phase 11.8 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow GC + SDK propagation because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)*
653
666
654
667
MVCC log-record WAL frame format (the deferred 11.4 piece). Commit appends log records pre-`save_database`. Reopen replays log records into `MvStore`. Checkpoint drains `MvStore` versions back into the pager (so `Mvcc → Wal` becomes legal once the store is empty). Crash-recovery test: kill mid-commit between log-record append and version-chain push; reopen; verify the committed transaction is visible and the half-written one is not.
Each secondary-index entry becomes its own `RowVersion`. Turso explicitly punted on this; SQLRite's v0 will reject `CREATE INDEX` while `journal_mode = mvcc`.
### Phase 11.10 — Multi-handle SDK shape + REPL `.spawn`*(planned, was plan-doc 11.8's other half)*
661
674
662
-
Surface `Busy` / `BusySnapshot`through the FFI shim and each language SDK. New REPL `.spawn` meta-command + new "N concurrent writers" benchmark workload.
675
+
Expose `Connection::connect()`through the FFI + each SDK so Python / Node / Go callers can mint sibling handles, plus a new REPL `.spawn` meta-command. Without this, the 11.7 retry-error machinery can't actually be exercised end-to-end through an SDK (each SDK `connect()` builds an independent DB). Also adds the "N concurrent writers" benchmark workload.
0 commit comments