feat(engine): Phase 11.6 MVCC garbage collection (SQLR-22) (#127)

Bounds in-memory MvStore growth. Without GC, every committed
version stays forever in the in-memory chain — a memory leak
that grows linearly with commits. Per-commit sweep + an
explicit Connection::vacuum_mvcc() for full-store drains
together cover the v0 model.

Promoted ahead of plan-doc 11.5 (checkpoint integration)
because durability already works through the legacy
save_database mirror (11.4). The unbounded-MvStore-growth gap
was the next concrete user-impact concern after 11.5 closed
the snapshot-read hole. Plan-doc 11.5 (checkpoint) → roadmap
11.7; plan-doc 11.6 (GC) → roadmap 11.6 (this slice).

New on MvStore:
- active_watermark() — returns min_active_begin_ts.unwrap_or
  (u64::MAX). Versions whose end is > watermark must be kept;
  ≤ watermark are reclaimable.
- gc_chain(row_id, watermark) — sweeps one row's chain,
  reclaims superseded versions, drops the row from the outer
  map if its chain becomes empty.
- gc_all(watermark) — full-store sweep. Snapshots row keys
  upfront so the outer map lock isn't held across per-chain
  locks.

Connection::commit_concurrent gains a per-commit GC sweep on
the write-set's chains. Crucially drops the TxHandle FIRST so
its begin_ts exits the active-tx registry — otherwise the
watermark would stay pinned to our own begin_ts and we'd
preserve every version we just wrote. With the handle dropped,
the watermark advances naturally and the sweep can reclaim
aggressively.

New Connection::vacuum_mvcc() public method runs a full-store
sweep at the current watermark. Returns the count reclaimed.
Safe to call regardless of journal_mode (no-op on Wal-mode
databases). The "vacuum the whole store" escape hatch for
memory-pressure workloads or tests that want a deterministic
baseline.

Tests (11 new):
- 7 in mvcc::store::tests:
  - active_watermark_is_max_with_no_readers
  - active_watermark_tracks_oldest_in_flight_tx
  - gc_chain_reclaims_versions_below_watermark
  - gc_chain_drops_empty_chain_from_map
  - gc_all_sweeps_every_row
  - gc_preserves_versions_visible_to_active_readers
  - gc_keeps_chain_bounded_under_repeated_updates
- 4 in connection::tests (integration):
  - repeated_updates_keep_chain_bounded_when_no_readers (50
    sequential UPDATEs leave 1 version in the store, not 50)
  - gc_preserves_versions_visible_to_active_reader
  - vacuum_mvcc_is_a_noop_on_wal_database
  - vacuum_mvcc_reclaims_everything_with_no_active_readers

650/650 workspace tests pass (was 639 + 11). fmt + clippy +
doc clean on changed files. No file-format change.

Docs: roadmap.md marks 11.5 ✅ and adds 11.6 🚧 with the
renumbering note (plan-doc 11.5 → roadmap 11.7, plan-doc 11.6
→ roadmap 11.6). design-decisions.md gets a 12f entry on the
per-commit + explicit-vacuum trade-offs and the TxHandle drop
ordering subtlety. embedding.md gains a "Memory bounding"
note. _index.md project-state line picks up the 11.6 summary.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: joaoh82

docs/_index.md

@@ -54,7 +54,7 @@ As of May 2026, SQLRite has:
 - Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
 - SQL surface + DX follow-ups (Phase 9 complete, v0.2.0 → v0.9.1): DDL completeness — `DEFAULT`, `DROP TABLE` / `DROP INDEX`, `ALTER TABLE` (9a); free-list + manual `VACUUM` (9b) + auto-VACUUM (9c); `IS NULL` / `IS NOT NULL` (9d); `GROUP BY` + aggregates + `DISTINCT` + `LIKE` + `IN` (9e); four flavors of `JOIN` — INNER, LEFT, RIGHT, FULL OUTER (9f); prepared statements + `?` parameter binding with a per-connection LRU plan cache (9g); HNSW probe widened to cosine + dot via `WITH (metric = …)` (9h); `PRAGMA` dispatcher with the `auto_vacuum` knob (9i)
 - Benchmarks against SQLite + DuckDB (Phase 10 complete, SQLR-4 / SQLR-16): twelve-workload bench harness with a pluggable `Driver` trait, criterion-driven, pinned-host runs published. See [`docs/benchmarks.md`](benchmarks.md).
-- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.4: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, and `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes go through commit-time validation against `MvStore` and abort with `SQLRiteError::Busy` on row-level write-write conflict; the four plan-required write-path tests pass. **11.5 snapshot-isolated reads via `Statement::query`: shipped on this branch.** `Connection.concurrent_tx` is now `Mutex<Option<…>>`; a new `with_snapshot_read` helper routes prepare()/query() through the BEGIN-time snapshot — closes the most user-visible 11.4 limitation. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
+- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.5: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, and `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes go through commit-time validation against `MvStore` and abort with `SQLRiteError::Busy` on row-level write-write conflict. Reads via `Statement::query` see the BEGIN-time snapshot. **11.6 garbage collection: shipped on this branch.** Per-commit GC sweep on the write-set's chains + a new `Connection::vacuum_mvcc()` for explicit full-store drains; bounds the in-memory version chain growth that 11.4–11.5 left unbounded. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
 - A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)
 
 See the [Roadmap](roadmap.md) for the full phase plan.

docs/design-decisions.md

@@ -229,7 +229,26 @@ Each statement inside the transaction runs against the working `tables` clone vi
 
 **The scope-guard pattern in `with_snapshot_read`.** The swap mutates `db.tables`. If the caller's closure panics mid-read, leaving `db.tables` pointing at the transaction's private clone would catastrophically corrupt every other handle's view of the database. The helper installs a `Drop` guard that unswaps on unwind; on the happy path the guard is disarmed and the unswap runs in the explicit code path so the borrow checker can see the field accesses are disjoint.
 
-**Plan-doc reference.** [`concurrent-writes-plan.md`](concurrent-writes-plan.md) §3.4 (connection model), §4.4 (read protocol — `MvStore`-backed reads in 11.6+).
+**Plan-doc reference.** [`concurrent-writes-plan.md`](concurrent-writes-plan.md) §3.4 (connection model), §4.4 (read protocol — `MvStore`-backed reads in 11.7+).
+
+---
+
+### 12f. `MvStore` GC sweeps per-commit, with `Connection::vacuum_mvcc` for explicit drains (Phase 11.6)
+
+**Decision.** Every successful `BEGIN CONCURRENT` commit ends with a per-commit GC sweep over the rows the transaction wrote. The sweep walks each row's chain and reclaims versions whose `end` timestamp is at or below the [`MvStore::active_watermark`](../src/mvcc/store.rs) (the smallest `begin_ts` across the active-tx registry, or `u64::MAX` when nothing is in flight). The latest version (`end == None`) and any in-flight version are always kept. An explicit [`Connection::vacuum_mvcc`](../src/connection.rs) method runs the same sweep across every row in the store.
+
+**Why per-commit + explicit, not periodic / background.** Three reasons:
+1. The per-commit sweep covers the rows we most care about — the ones we just modified, whose chains just grew. No stale versions accumulate on a hot row under repeated updates.
+2. Background-thread GC adds an extra runtime mode (interval pragma, scheduler hooks, shutdown coordination) for a small win in v0. The per-commit sweep is amortised across the work the engine is already doing, and `vacuum_mvcc` covers the edge cases.
+3. Tests + memory-pressure debug paths want a deterministic "drain to nothing" lever; the explicit method is that lever.
+
+**Why drop the `TxHandle` before the sweep.** The handle holds the transaction's own `begin_ts` in the active-tx registry. Sweeping while the handle is live would pin the watermark to our own `begin_ts` and preserve every version we just wrote (because their `end` timestamps would be at or above our own `begin_ts`). Dropping the handle first lets the watermark advance to the next-oldest active reader (or `u64::MAX` when we were the only one), so the sweep can reclaim aggressively.
+
+**Why drop empty rows from the outer map.** Long-running sessions that delete + reinsert across many distinct rowids would otherwise accumulate empty `Arc<RwLock<Vec<RowVersion>>>` chains. The sweep checks under both locks (outer map + chain) before removing, so it can't race with a `push_committed` about to add a new version.
+
+**Why the watermark uses `u64::MAX` rather than `clock.now()` when no readers are active.** Both work; `u64::MAX` is cheaper (no clock read) and produces the same outcome (every superseded version is reclaimable).
+
+**Plan-doc reference.** [`concurrent-writes-plan.md`](concurrent-writes-plan.md) §4.7 (garbage collection), §8 (memory growth as a known risk).
 
 ---
 

docs/embedding.md

@@ -124,11 +124,13 @@ fn transfer(primary: &mut Connection, src: i64, dst: i64, amount: i64)
 
 The retryable-error branch is the headline new flow: pick a backoff policy that suits your workload (constant, exponential, jittered) and call the same closure again. `SQLRiteError::is_retryable()` covers both `Busy` and `BusySnapshot` so callers don't have to match each variant individually.
 
-**What's still ahead** (11.6+):
+**Memory bounding.** Every successful commit triggers a per-row GC sweep over the write-set's chains, reclaiming versions no in-flight reader can possibly see anymore. For workloads where you want a deterministic full drain (memory-pressure testing, debug snapshots), call `conn.vacuum_mvcc()` — returns the count of versions reclaimed across the whole store. Both paths are correct against in-flight readers: a reader holding `BEGIN CONCURRENT; SELECT …` keeps every version its `begin_ts` snapshot needs.
 
-- Reads inside the transaction now see the BEGIN-time snapshot via both `Connection::execute("SELECT…")` and `Statement::query()` / `query_with_params()` — the prepare/query gap that 11.4 left open closed in 11.5.
+**What's still ahead** (11.7+):
+
+- Reads inside the transaction see the BEGIN-time snapshot via both `Connection::execute("SELECT…")` and `Statement::query()` / `query_with_params()` — the prepare/query gap that 11.4 left open closed in 11.5.
 - DDL inside `BEGIN CONCURRENT` is rejected with a typed error.
-- The transaction's write-set persists only via the legacy `Database::tables` mirror — a crash mid-transaction loses everything (correct behaviour, the transaction never committed). Phase 11.6 introduces an MVCC log-record WAL frame so `BEGIN CONCURRENT` writes become durable through `MvStore` itself.
+- The transaction's write-set persists only via the legacy `Database::tables` mirror — a crash mid-transaction loses everything (correct behaviour, the transaction never committed). Phase 11.7 introduces an MVCC log-record WAL frame so `BEGIN CONCURRENT` writes become durable through `MvStore` itself.
 
 ### What's deferred
 

docs/roadmap.md

@@ -594,7 +594,7 @@ Lift SQLRite past SQLite's single-writer ceiling with multi-version concurrency
 [`sqlrite::mvcc`](../src/mvcc/) module:
 
 - `MvccClock` — process-wide monotonic `u64` over `AtomicU64`. `tick()` hands out begin- / commit-timestamps; `now()` reads the high-water without advancing it; `observe(value)` advances the clock to `value` if greater (used at WAL replay).
-- `ActiveTxRegistry` — `Mutex<BTreeMap>` over in-flight transactions. `register(&clock)` allocates a `TxId`, snapshots `begin_ts`, and returns a RAII `TxHandle`; `min_active_begin_ts()` answers Phase 11.6 GC's "what's still possibly visible" question.
+- `ActiveTxRegistry` — `Mutex<BTreeMap>` over in-flight transactions. `register(&clock)` allocates a `TxId`, snapshots `begin_ts`, and returns a RAII `TxHandle`; `min_active_begin_ts()` is the GC watermark Phase 11.6 reads on every commit + on `Connection::vacuum_mvcc`.
 - `TxId` newtype + `TxTimestampOrId` tagged union — defined now so 11.4 can plug in without re-litigating the type shape.
 
 WAL format bumps **v1 → v2**: bytes 24..32 of the WAL header (previously reserved-zero) now carry the persisted `clock_high_water` `u64`. v1 WALs open cleanly — those zero bytes read as "clock never advanced" — and the next checkpoint rewrites the header at v2. No offline upgrade step. `Wal::set_clock_high_water` / `Wal::clock_high_water` accessors expose the field; the setter rejects regressions with a typed error.
@@ -622,25 +622,36 @@ The headline slice. Multiple sibling `Connection`s can each hold their own open
 **Known limitations carried forward (most resolved in 11.5):**
 
 - ~~Reads via `Statement::query` / `Statement::query_with_params` bypass the swap.~~ ✅ Fixed in 11.5 — `Connection.concurrent_tx` is now `Mutex<Option<…>>` and a new `with_snapshot_read` helper threads the swap through `&self`.
-- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.6 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror.
+- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.7 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.)
 - `AUTOINCREMENT` inside `BEGIN CONCURRENT` isn't explicitly rejected; the v0 deep-clone-snapshot model handles concurrent INSERTs by isolating each tx's `last_rowid` bumps to its private snapshot, so two concurrent INSERTs on an `AUTOINCREMENT` column may collide at COMMIT and surface as `Busy`. Adopting the plan's "reject AUTOINCREMENT under MVCC" gate is a clean follow-up.
 - Tables touched by `BEGIN CONCURRENT` writes can't carry FTS or HNSW indexes today — `restore_row` only maintains B-tree secondary indexes. Concurrent-tx tests don't exercise FTS / HNSW, but a runtime guard would surface this with a clear error rather than producing inconsistent indexes.
 
-### 🚧 Phase 11.5 — Snapshot-isolated reads via `Statement::query` *(in progress, slotted ahead of plan-doc 11.5 checkpoint work because the prepare/query gap was the most user-visible 11.4 limitation)*
+### ✅ Phase 11.5 — Snapshot-isolated reads via `Statement::query` *(slotted ahead of plan-doc 11.5 checkpoint work because the prepare/query gap was the most user-visible 11.4 limitation)*
 
 `Connection.concurrent_tx` is now `Mutex<Option<ConcurrentTx>>` (was plain `Option`). A new `with_snapshot_read` helper takes `&self`, locks `concurrent_tx`, then locks the database, and — when a tx is open — swaps the tx's private cloned `tables` in for the duration of the read closure (with a scope-guarded unswap so a panic inside the closure can't strand the database). [`Statement::query`] and [`Statement::query_with_params`] route through this helper so the prepared-statement path now sees the same BEGIN-time snapshot the `execute("SELECT…")` path already saw in 11.4.
 
-Lock order is consistently `concurrent_tx → inner` across every code path; deadlock-free by construction. `Connection` is still `Send + Sync`. The four 11.4 plan-required tests still pass; new tests cover snapshot reads via prepare/query, read-your-writes within a tx, parameter-bound SELECT through the snapshot, and snapshot consistency across concurrent sibling commits.
+Lock order is consistently `concurrent_tx → inner` across every code path; deadlock-free by construction. `Connection` is still `Send + Sync`.
 
-This was renumbered out of plan-doc order: the plan-doc had 11.5 as checkpoint integration, but that's a much larger slice and the prepare/query-bypass-the-swap gap was a real correctness hole for users hitting `BEGIN CONCURRENT`. Plan-doc 11.5 (checkpoint) becomes 11.6 here; plan-doc 11.6 (GC) becomes 11.7; etc.
+This was renumbered out of plan-doc order: the plan-doc had 11.5 as checkpoint integration, but that's a much larger slice and the prepare/query-bypass-the-swap gap was a real correctness hole for users hitting `BEGIN CONCURRENT`. Plan-doc 11.5 (checkpoint) → roadmap 11.7; plan-doc 11.6 (GC) → roadmap 11.6 (this one).
 
-### Phase 11.6 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5")*
+### 🚧 Phase 11.6 — Garbage collection *(in progress, plan-doc "Phase 10.6"; promoted ahead of plan-doc 11.5 because unbounded `MvStore` growth was the next concrete user-impact concern after 11.5 closed the snapshot-read gap)*
 
-MVCC log-record WAL frame format (the deferred 11.4 piece). Commit appends log records pre-`save_database`. Reopen replays log records into `MvStore`. Checkpoint drains `MvStore` versions back into the pager (so `Mvcc → Wal` becomes legal once the store is empty). Crash-recovery test: kill mid-commit between log-record append and version-chain push; reopen; verify the committed transaction is visible and the half-written one is not.
+Bounds in-memory growth of the [`MvStore`](../src/mvcc/store.rs) version chains. Without this, every committed version stays forever in the in-memory chain — a memory leak that grows linearly with commits.
+
+- `MvStore::active_watermark()` returns the GC watermark — the smallest `begin_ts` across the active-tx registry, or `u64::MAX` when nothing is in flight. Versions whose committed `end` timestamp is `<= watermark` are reclaimable: no reader's `begin_ts` can fall in the half-open `[begin, end)` interval that snapshot-isolation visibility requires.
+- `MvStore::gc_chain(row_id, watermark)` reclaims one row's superseded versions (kept: latest version with `end == None`, in-flight versions, and any committed version still possibly visible to a reader). Returns the number of versions dropped. Drops the row from the outer map entirely if its chain becomes empty so long-running sessions don't leak per-row entries.
+- `MvStore::gc_all(watermark)` sweeps every row in one pass; returns total versions reclaimed. Snapshots the row keys upfront so the outer map lock isn't held across per-chain locks.
+- [`Connection::commit_concurrent`](../src/connection.rs) gains a per-commit GC sweep on the write-set's chains. Drops the `tx` `TxHandle` *first* so its `begin_ts` exits the registry — otherwise the watermark is still pinned to our own `begin_ts` and we'd preserve versions we're free to reclaim. Cheap (sweeps only the rows this transaction wrote), and runs on every successful commit.
+- New `Connection::vacuum_mvcc()` method runs a full-store sweep at the current watermark. Returns the version count reclaimed. The "vacuum the whole store" escape hatch for memory-pressure workloads or tests that want a deterministic baseline. Safe to call regardless of `journal_mode` (a no-op `Wal`-mode database returns 0).
+
+**What 11.6 doesn't yet do:**
 
-### Phase 11.7 — Garbage collection *(planned, plan-doc "Phase 10.6")*
+- No background GC thread or `PRAGMA mvcc_gc_interval_ms`. Per-commit sweep + explicit `vacuum_mvcc()` cover the v0 model; the periodic-sweep variant lands as a follow-up if profiles show it's needed.
+- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.7.
 
-Per-commit sweep over the write-set's chains, plus a background sweep behind `PRAGMA mvcc_gc_interval_ms`. `min_active_begin_ts` watermark already lives in `ActiveTxRegistry`.
+### Phase 11.7 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow GC because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)*
+
+MVCC log-record WAL frame format (the deferred 11.4 piece). Commit appends log records pre-`save_database`. Reopen replays log records into `MvStore`. Checkpoint drains `MvStore` versions back into the pager (so `Mvcc → Wal` becomes legal once the store is empty). Crash-recovery test: kill mid-commit between log-record append and version-chain push; reopen; verify the committed transaction is visible and the half-written one is not.
 
 ### Phase 11.8 — Indexes under MVCC *(deferred-by-design, plan-doc "Phase 10.7")*