Skip to content

Commit 0d22d8c

Browse files
joaoh82claude
andauthored
docs: Phase 11.12 — promote concurrent-writes-plan.md to canonical reference (SQLR-22) (#132)
Closes plan-doc §10.9 (the docs slice). Phase 11 (concurrent writes via MVCC + BEGIN CONCURRENT, SQLR-22) is now shipped end-to-end through 11.12 — every shipped sub-phase had been updating the plan-doc piecemeal; this slice promotes the user-facing surface into one canonical reference and reshuffles cross-refs so callers stop bouncing through the historical design record to find the SQL grammar or the retry-loop shape. New docs: - docs/concurrent-writes.md — canonical user-facing reference: TL;DR, conceptual model (version chains, logical clock, commit validation, snapshot reads), SQL surface (PRAGMA journal_mode + BEGIN CONCURRENT + COMMIT/ROLLBACK rules), embedding API (sibling handles + retry loop + vacuum_mvcc), per-SDK error mapping table, REPL .spawn/.use/ .conns demo, error semantics, durability + recovery (WAL log records, v3 format, clock seeding), limitations, references. New example: - examples/rust/concurrent_writers.rs — registered as `cargo run --example concurrent_writers`. Two sibling Connections via Connection::connect; runs (1) disjoint-row commits both succeed and (2) interleaved same-row commits where one wins and the other catches Busy + retries. Verified end-to-end: alice ends at 100 + 10 + 5 + 50 = 165. Updated cross-refs: - docs/concurrent-writes-plan.md — redirect banner at the top pointing readers to the canonical doc; "Status" line updated to "shipped end- to-end through 11.11a"; plan-doc §10.9 entry annotated with what landed in 11.12. - docs/_index.md — Phase 11 summary refreshed to "shipped end-to-end through 11.11a + 11.12 docs sweep"; new "Concurrent writes" entry in the "Using SQLRite as a library" section. - docs/supported-sql.md — PRAGMA journal_mode and BEGIN CONCURRENT sections gain a "see concurrent-writes.md for the full reference" callout; the "Not yet supported" entry for BEGIN CONCURRENT removed (it ships). - docs/embedding.md — Phase 11.4 section refreshed with a pointer at the canonical doc + the new example; "what's still ahead" turned into "what shipped after 11.4" with 11.5 → 11.11a captured. - docs/roadmap.md — active-frontier blurb expanded to enumerate every shipped sub-phase + the explicitly parked follow-ups; Phase 11.12 entry promoted to ✅ shipped. - docs/design-decisions.md — Phase 11 §12a–§12h gain a leading banner pointing readers at the canonical doc for the user-facing view. - examples/README.md — concurrent_writers entry added alongside quickstart + hybrid-retrieval. Workspace: 615/615 Rust tests still pass; cargo fmt + clippy + doc all clean. The new example builds + runs successfully. What's still open in Phase 11: - 11.10 — Indexes under MVCC (deferred-by-design, plan-doc §10.7) - 11.11b — "N concurrent writers" bench workload + Go SDK cross-pool sibling shape (carved out from the original 11.11) - Checkpoint-drain follow-up (parked half of 11.9 — enables set_journal_mode(Mvcc → Wal) once MvStore can be drained) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 4495a68 commit 0d22d8c

10 files changed

Lines changed: 490 additions & 22 deletions

Cargo.toml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,15 @@ path = "examples/rust/quickstart.rs"
7373
name = "hybrid-retrieval"
7474
path = "examples/hybrid-retrieval/hybrid_retrieval.rs"
7575

76+
# Phase 11.12 — BEGIN CONCURRENT retry-loop demo. Mints a sibling
77+
# Connection via `Connection::connect`, runs two concurrent
78+
# transactions (first disjoint, then same-row), and surfaces the
79+
# Busy / retry path. Run with `cargo run --example concurrent_writers`.
80+
# See `docs/concurrent-writes.md` for the conceptual walkthrough.
81+
[[example]]
82+
name = "concurrent_writers"
83+
path = "examples/rust/concurrent_writers.rs"
84+
7685
[features]
7786
# Default build includes everything: the REPL binary (cli) and
7887
# POSIX/Windows advisory file locks on the Pager (file-locks).

docs/_index.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ A small, hand-written guide to the SQLRite codebase — how it's structured, how
1616
## Using SQLRite as a library
1717

1818
- [Embedding](embedding.md) — the public `Connection` / `Statement` / `Rows` API (Phase 5a) and where the non-Rust SDKs plug in (Phase 5b – 5g)
19-
- [`examples/`](../examples/) — runnable Rust quickstart (`cargo run --example quickstart`); language-specific subdirectories fill in as each 5x sub-phase lands
19+
- [Concurrent writes — MVCC + `BEGIN CONCURRENT`](concurrent-writes.md) — Phase 11 canonical reference: SQL surface, embedding API, SDK error mapping, REPL meta-commands, durability story, limitations. Design rationale lives in the [historical plan-doc](concurrent-writes-plan.md).
20+
- [`examples/`](../examples/) — runnable Rust quickstart (`cargo run --example quickstart`) + concurrent-writers retry-loop demo (`cargo run --example concurrent_writers`); language-specific subdirectories fill in as each 5x sub-phase lands
2021

2122
## Phase 7 — AI-era extensions
2223

@@ -54,7 +55,7 @@ As of May 2026, SQLRite has:
5455
- Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
5556
- SQL surface + DX follow-ups (Phase 9 complete, v0.2.0 → v0.9.1): DDL completeness — `DEFAULT`, `DROP TABLE` / `DROP INDEX`, `ALTER TABLE` (9a); free-list + manual `VACUUM` (9b) + auto-VACUUM (9c); `IS NULL` / `IS NOT NULL` (9d); `GROUP BY` + aggregates + `DISTINCT` + `LIKE` + `IN` (9e); four flavors of `JOIN` — INNER, LEFT, RIGHT, FULL OUTER (9f); prepared statements + `?` parameter binding with a per-connection LRU plan cache (9g); HNSW probe widened to cosine + dot via `WITH (metric = …)` (9h); `PRAGMA` dispatcher with the `auto_vacuum` knob (9i)
5657
- Benchmarks against SQLite + DuckDB (Phase 10 complete, SQLR-4 / SQLR-16): twelve-workload bench harness with a pluggable `Driver` trait, criterion-driven, pinned-host runs published. See [`docs/benchmarks.md`](benchmarks.md).
57-
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.9: shipped.** Engine + SDK error propagation: `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, `ConcurrentTx`, and the `MvccCommitBatch` / `MvccLogRecord` WAL codec. WAL header v1 → v2 persisted the clock high-water mark; **v2 → v3 (11.9)** adds typed MVCC log-record frames. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes commit-validate against `MvStore`, abort with `SQLRiteError::Busy`, and now also append an MVCC log-record frame to the WAL — covered by the same fsync as the legacy page commit. Reopen replays those frames into `MvStore` and seeds `MvccClock` past the highest committed `commit_ts`, so the MVCC conflict-detection window survives a process restart. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `vacuum_mvcc()` bound version-chain growth. C FFI / Python / Node / Go all propagate `Busy` / `BusySnapshot` as typed retryable errors; the FFI's `sqlrite_connect_sibling`, Python's `Connection.connect()`, and Node's `db.connect()` mint sibling handles that share backing state. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
58+
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is **shipped end-to-end through 11.11a** plus the 11.12 docs sweep — a small set of follow-ups (checkpoint-drain to enable `Mvcc → Wal` downgrade; indexes under MVCC; the "N concurrent writers" benchmark workload) remain explicitly parked. `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, `ConcurrentTx`, and the `MvccCommitBatch` / `MvccLogRecord` WAL codec. WAL header v1 → v2 persisted the clock high-water mark; v2 → v3 added typed MVCC log-record frames. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes commit-validate against `MvStore`, abort with `SQLRiteError::Busy`, and append a typed MVCC log-record frame to the WAL — covered by the same fsync as the legacy page commit. Reopen replays those frames into `MvStore` and seeds `MvccClock` past the highest committed `commit_ts`, so the MVCC conflict-detection window survives a process restart. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `vacuum_mvcc()` bound version-chain growth. C FFI / Python / Node / Go propagate `Busy` / `BusySnapshot` as typed retryable errors; the FFI's `sqlrite_connect_sibling`, Python's `Connection.connect()`, and Node's `db.connect()` mint sibling handles that share backing state. The `sqlrite` REPL ships `.spawn` / `.use` / `.conns` for interactive demos. **User-facing reference:** [`docs/concurrent-writes.md`](concurrent-writes.md); runnable example at [`examples/rust/concurrent_writers.rs`](../examples/rust/concurrent_writers.rs). Original design proposal: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
5859
- A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)
5960

6061
See the [Roadmap](roadmap.md) for the full phase plan.

docs/concurrent-writes-plan.md

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,16 @@
11
# Concurrent writes plan — MVCC + `BEGIN CONCURRENT`
22

3-
**Status:** proposal, not yet scheduled. Drafted 2026-05-07.
3+
> 📘 **Looking for the user-facing reference?** This is the original
4+
> design proposal, kept as the historical record of the decisions
5+
> that shaped Phase 11. For the shipped surface — SQL, embedding API,
6+
> SDK error mapping, REPL meta-commands, durability story,
7+
> limitations — read [**`concurrent-writes.md`**](concurrent-writes.md)
8+
> first; come back here when you want the *why* and the
9+
> sequencing discussion.
10+
11+
**Status:** shipped end-to-end through Phase 11.11a (May 2026); a small set of follow-ups remain explicitly parked — see the [roadmap](roadmap.md#phase-11--concurrent-writes-via-mvcc--begin-concurrent-sqlr-22-in-flight--see-concurrent-writes-planmd). Drafted 2026-05-07.
412
**Inspiration:** [Turso](https://turso.tech) — a SQLite-compatible engine, written in Rust, that implements multi-version concurrency control to lift SQLite's single-writer ceiling. See [`turso/core/mvcc/`](https://github.com/tursodatabase/turso/tree/main/core/mvcc) and the [Turso concurrent-writes docs](https://docs.turso.tech/tursodb/concurrent-writes).
5-
**Tracks:** SQLR-?? (Marvin) — to be filed alongside this doc.
13+
**Tracks:** [SQLR-22](https://app.marvinapp.io/) (Marvin).
614

715
This document proposes adding **multi-version concurrency control (MVCC)** and a **`BEGIN CONCURRENT`** transaction mode to SQLRite, enabling multiple writers in the same process to make progress in parallel under snapshot isolation, with row-level write-write conflict detection at commit. It is intentionally a *plan* — there is no code yet.
816

@@ -296,9 +304,11 @@ Index maintenance under MVCC is hard enough that Turso explicitly punted on it.
296304

297305
### Phase 10.9 — Docs
298306

299-
- Promote this plan to `docs/concurrent-writes.md` (the canonical user-facing reference), keeping `concurrent-writes-plan.md` as the historical design document.
300-
- Update [roadmap.md](roadmap.md), [`docs/_index.md`](_index.md), [supported-sql.md](supported-sql.md), [embedding.md](embedding.md), [design-decisions.md](design-decisions.md).
301-
- Add a worked example under `examples/rust/concurrent_writers.rs`.
307+
> **Status (roadmap 11.12 — May 2026):** Shipped. The canonical user-facing reference at [`docs/concurrent-writes.md`](concurrent-writes.md) covers the SQL surface, embedding API, SDK error mapping, REPL meta-commands, durability story, and limitations as of Phase 11.11a. This plan-doc is now the historical record. Cross-references in `_index.md`, `supported-sql.md`, `embedding.md`, and `design-decisions.md` point at the canonical doc; a runnable example lives at [`examples/rust/concurrent_writers.rs`](../examples/rust/concurrent_writers.rs).
308+
309+
- Promote this plan to `docs/concurrent-writes.md` (the canonical user-facing reference), keeping `concurrent-writes-plan.md` as the historical design document. **(Shipped — 11.12.)**
310+
- Update [roadmap.md](roadmap.md), [`docs/_index.md`](_index.md), [supported-sql.md](supported-sql.md), [embedding.md](embedding.md), [design-decisions.md](design-decisions.md). **(Shipped — 11.12.)**
311+
- Add a worked example under `examples/rust/concurrent_writers.rs`. **(Shipped — 11.12.)**
302312

303313
---
304314

0 commit comments

Comments
 (0)