feat(fts): 8f — final docs sweep for full-text search (#83)

Phase 8 sub-phase 8f per docs/phase-8-plan.md. Closes the
docs gap left by 8a–8e. No code changes; pure documentation.

New:

- docs/fts.md — canonical FTS reference. Mirrors docs/ask.md's
  shape: TOC, what FTS does, quickstart, SQL surface
  (CREATE INDEX … USING fts, fts_match, bm25_score), hybrid
  retrieval, tokenizer rules, BM25 parameters, lifecycle
  (INSERT / DELETE / UPDATE), persistence + on-demand v4→v5
  bump, optimizer hook, per-surface usage (REPL / Rust / MCP /
  SDKs / desktop), limitations, and see-also links. Covers
  every Q1–Q10 design decision with cross-references back to
  the Phase 8 plan.

Updates:

- docs/supported-sql.md — new "FTS indexes (Phase 8)" subsection
  under CREATE INDEX, plus fts_match + bm25_score rows in the
  Built-in functions table, plus an "FTS query patterns" code
  block alongside the JSON example.
- docs/architecture.md — adds src/sql/fts/ to the engine module
  map; updates the executor entry to mention the FTS probe
  shortcut (alongside HNSW); updates the on-disk storage entry
  to mention FTS posting lists + the on-demand v5 bump.
- docs/file-format.md — bumps the "current on-disk format" line
  to "v4 by default, v5 on-demand"; updates the page-0 header
  diagram; adds KIND_FTS_POSTING (0x06) to the cell-tag table;
  documents the full FTS posting cell layout (cell_id, term_len,
  term, count, (rowid, value) pairs); adds a v5 entry to the
  Format evolution section.
- docs/sql-engine.md — adds an "Optimizer hooks" subsection
  covering both try_hnsw_probe and try_fts_probe with their
  recognized shapes and fall-through rules.
- docs/mcp.md — renames "seven tools" → "eight tools" (table,
  table-of-contents, mcp-inspector callout); adds a
  bm25_search section symmetric with vector_search.
- docs/smoke-test.md — adds section 1.9b: CREATE INDEX … USING
  fts, fts_match WHERE filter, bm25_score top-k.
- docs/_index.md — moves Phase 8 from "Future work" to its own
  top-level section under the docs index; updates the project-
  state bullet to reflect Phase 8 complete.
- docs/roadmap.md — Phase 8 section moves from "active frontier
  draft" to "✅ complete" with a per-sub-phase ledger (8a-8f),
  PR numbers, and one-paragraph summaries of what each shipped.
  The top-of-file frontier line updates to "v0.2.0 release" as
  the next milestone.

Total docs delta: +175 lines net across 8 files plus the new
fts.md (~290 lines).

Closes Phase 8.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: joaoh82

docs/_index.md

@@ -22,7 +22,11 @@ A small, hand-written guide to the SQLRite codebase — how it's structured, how
 
 - [Ask — natural-language → SQL](ask.md) — the canonical reference for the `ask()` feature across every product surface (REPL, desktop, Rust library, Python / Node / Go / WASM SDKs, MCP server); env vars, defaults, prompt caching, security
 - [Ask backend proxy templates](ask-backend-examples.md) — copy-paste backend examples for the WASM SDK's split design: Cloudflare Workers, Vercel Edge, Deno Deploy, Firebase Functions, AWS Lambda, Express, pure Node
-- [MCP server (`sqlrite-mcp`)](mcp.md) — Phase 7h: SQLRite as a Model Context Protocol stdio server. Wiring into Claude Code / Cursor / `mcp-inspector`; the seven tools (`list_tables`, `describe_table`, `query`, `execute`, `schema_dump`, `vector_search`, `ask`); read-only mode; the JSON-RPC wire format
+- [MCP server (`sqlrite-mcp`)](mcp.md) — Phase 7h + 8e: SQLRite as a Model Context Protocol stdio server. Wiring into Claude Code / Cursor / `mcp-inspector`; the eight tools (`list_tables`, `describe_table`, `query`, `execute`, `schema_dump`, `vector_search`, `bm25_search`, `ask`); read-only mode; the JSON-RPC wire format
+
+## Phase 8 — Full-text search + hybrid retrieval
+
+- [FTS — full-text search + hybrid retrieval](fts.md) — the canonical reference for `CREATE INDEX … USING fts`, the `fts_match` / `bm25_score` scalar functions, the `try_fts_probe` optimizer hook, hybrid retrieval via raw arithmetic with `vec_distance_cosine`, persistence + the on-demand v4 → v5 file-format bump, and the `bm25_search` MCP tool
 
 ## Internals
 
@@ -42,11 +46,10 @@ As of May 2026, SQLRite has:
 - WAL-backed persistence with crash-safe checkpointing, shared/exclusive lock modes, and real `BEGIN` / `COMMIT` / `ROLLBACK` transactions (Phase 4 complete)
 - A stable public Rust embedding API plus C FFI shim and SDKs for Python, Node.js, Go, and WASM (Phase 5 complete except the optional 5f crate-polish task)
 - A Tauri 2.0 + Svelte desktop app (Phase 2.5 complete)
-- AI-era extensions across the product surface (Phase 7 complete except FTS): VECTOR columns + HNSW indexes (7a-7d), JSON columns (7e), the `ask()` natural-language → SQL family across the REPL / desktop / Rust / Python / Node / Go / WASM (7g.1-7g.7), and the [`sqlrite-mcp`](mcp.md) Model Context Protocol server with seven tools including `ask` (7h + 7g.8)
+- AI-era extensions across the product surface (Phase 7 complete): VECTOR columns + HNSW indexes (7a-7d), JSON columns (7e), the `ask()` natural-language → SQL family across the REPL / desktop / Rust / Python / Node / Go / WASM (7g.1-7g.7), and the [`sqlrite-mcp`](mcp.md) Model Context Protocol server (7h + 7g.8)
+- Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
 - A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)
 
-The active frontier is **Phase 8 — Full-text search + hybrid retrieval** (the deferred 7f scope).
-
 See the [Roadmap](roadmap.md) for the full phase plan.
 
 ## Release engineering
@@ -58,7 +61,7 @@ See the [Roadmap](roadmap.md) for the full phase plan.
 ## Future work
 
 - [Phase 7 plan](phase-7-plan.md) — AI-era extensions (vector column type + HNSW, JSON, NL→SQL `ask()` API across REPL/library/SDKs/desktop/MCP, MCP server). **Implementation complete except 7f, which deferred to Phase 8.**
-- [Phase 8 plan](phase-8-plan.md) — Full-text search (FTS5-style BM25) + hybrid retrieval. The deferred 7f scope. **Draft 2026-05-03 — awaiting Q1–Q10 sign-off before implementation starts at sub-phase 8a.**
+- [Phase 8 plan](phase-8-plan.md) — Full-text search (FTS5-style BM25) + hybrid retrieval. The deferred 7f scope. **All six sub-phases (8a–8f) shipped.** Canonical reference: [`docs/fts.md`](fts.md).
 
 ## Conventions
 

docs/architecture.md

@@ -97,10 +97,11 @@ The engine never depends on the SDK crates; the SDK crates each depend on the en
 | [`src/error.rs`](../src/error.rs) | `SQLRiteError` (thiserror-derived), `Result<T>` alias, hand-rolled `PartialEq` that handles `io::Error` |
 | [`src/sql/mod.rs`](../src/sql/mod.rs) | `SQLCommand` classifier, `process_command` / `process_command_with_render` — the top-level entries that parse a SQL string and route to the right executor. Also triggers auto-save. **Never writes to stdout** — for SELECT statements, the rendered prettytable comes back inside `CommandOutput.rendered` so the REPL can print it (the engine itself doesn't); the SDK / FFI / MCP callers ignore it. |
 | [`src/sql/parser/`](../src/sql/parser/) | Takes a `sqlparser::ast::Statement` and produces internal query structs (`CreateQuery`, `InsertQuery`, `SelectQuery`) with only the fields we actually use |
-| [`src/sql/executor.rs`](../src/sql/executor.rs) | `execute_select`, `execute_delete`, `execute_update`, plus the shared expression evaluator `eval_expr` / `eval_predicate`. Also the bounded-heap top-k optimization (Phase 7c) and the HNSW probe shortcut (Phase 7d.2). |
+| [`src/sql/executor.rs`](../src/sql/executor.rs) | `execute_select`, `execute_delete`, `execute_update`, plus the shared expression evaluator `eval_expr` / `eval_predicate`. Also the bounded-heap top-k optimization (Phase 7c), the HNSW probe shortcut (Phase 7d.2), and the FTS probe shortcut (Phase 8b). |
 | [`src/sql/db/database.rs`](../src/sql/db/database.rs) | `Database`: table map + optional `source_path` + optional long-lived `Pager` + transaction-snapshot state |
 | [`src/sql/db/table.rs`](../src/sql/db/table.rs) | `Table`, `Column`, `Row`, `Value` (in-memory storage incl. VECTOR + JSON columns); helpers for row iteration (`rowids`, `get_value`, `set_value`, `delete_row`, `insert_row`) |
 | [`src/sql/hnsw.rs`](../src/sql/hnsw.rs) | Standalone HNSW algorithm — insert / search / layer assignment / beam search. Phase 7d.1. |
+| [`src/sql/fts/`](../src/sql/fts/) | Full-text search — standalone tokenizer, BM25 scorer, and in-memory `PostingList` inverted index. Wired into the executor via the `fts_match` / `bm25_score` scalar functions and the `try_fts_probe` optimizer hook. Phase 8a-8b; persistence in 8c. See [`docs/fts.md`](fts.md). |
 | [`src/sql/json.rs`](../src/sql/json.rs) | JSON column type + path-extraction functions (`json_extract`, `json_type`, `json_array_length`, `json_object_keys`). Phase 7e. |
 | [`src/sql/pager/`](../src/sql/pager/) | On-disk file format and I/O — see [file-format.md](file-format.md) and [pager.md](pager.md) for details. WAL + checkpointer + shared/exclusive lock modes (Phase 4a-4e) live here. |
 
@@ -127,7 +128,7 @@ Steps 1–7 are purely in-memory; step 8 is the only disk contact, and after the
 - **Planning**: intentionally not a thing yet. Execution is direct — a query plan is implicit in the executor code path.
 - **Execution**: `src/sql/executor.rs` walks the internal structs, drives reads against `Table`, and writes via `Table::set_value` / `insert_row` / `delete_row`.
 - **Storage (in memory)**: `src/sql/db/table.rs` — column-oriented `BTreeMap<rowid, value>` per column; indexes as separate `BTreeMap`s on UNIQUE/PK columns.
-- **Storage (on disk)**: `src/sql/pager/` — 4 KiB pages, real B-Tree per table (Phase 3d), secondary indexes (3e), HNSW indexes as their own page tree (7d.3), WAL + crash-safe checkpointer (4c-4d), shared/exclusive lock modes (4e).
+- **Storage (on disk)**: `src/sql/pager/` — 4 KiB pages, real B-Tree per table (Phase 3d), secondary indexes (3e), HNSW indexes as their own page tree (7d.3), FTS posting lists as their own page tree (8c, on-demand v5 file format), WAL + crash-safe checkpointer (4c-4d), shared/exclusive lock modes (4e).
 - **Persistence policy**: `src/sql/mod.rs::process_command` for when to auto-save; `src/sql/pager/mod.rs::save_database` for how. Inside a `BEGIN`/`COMMIT` block, auto-save is suppressed and changes accumulate against an in-memory snapshot — `COMMIT` flushes the whole batch in one WAL frame; `ROLLBACK` restores the snapshot.
 - **Error handling**: `src/error.rs` defines a single `SQLRiteError` enum used throughout, with `#[from]` conversions from `ParserError` and `io::Error`.
 

docs/file-format.md

@@ -4,7 +4,7 @@ A SQLRite database is a single file, by convention named `*.sqlrite`. The file i
 
 All multi-byte integers in this format are **little-endian**.
 
-The current on-disk format is **version 3** (Phase 3e). Files produced by earlier versions are rejected on open.
+The current on-disk format is **version 4** (Phase 7) by default, with **version 5** written on demand whenever an FTS index is attached to the database (Phase 8c). Decoders accept both v4 and v5; writers preserve the existing version on no-op resaves so a v4 database without FTS stays v4. Files produced by versions 1 – 3 are rejected on open.
 
 ## Page 0 — the database header
 
@@ -15,7 +15,7 @@ The first 4096 bytes of every file are the header page. Only the first 28 bytes
 │ offset │ length │ content                                         │
 ├────────┼────────┼─────────────────────────────────────────────────┤
 │     0  │   16   │ magic:  "SQLRiteFormat\0\0\0"                   │
-│    16  │    2   │ format version (u16 LE) = 3                     │
+│    16  │    2   │ format version (u16 LE) = 4 or 5                │
 │    18  │    2   │ page size      (u16 LE) = 4096                  │
 │    20  │    4   │ total page count (u32 LE), includes page 0      │
 │    24  │    4   │ root page of sqlrite_master (u32 LE)            │
@@ -25,7 +25,7 @@ The first 4096 bytes of every file are the header page. Only the first 28 bytes
 
 The magic string is 14 ASCII bytes (`SQLRiteFormat`) padded with two NUL bytes to fill 16 bytes. It's deliberately different from SQLite's `"SQLite format 3\0"` so the two formats can't be confused on inspection.
 
-`decode_header` in [`src/sql/pager/header.rs`](../src/sql/pager/header.rs) validates all three of (magic, format version, page size) on open. A wrong magic produces `not a SQLRite database`; a wrong version or page size produces `unsupported ...` errors.
+`decode_header` in [`src/sql/pager/header.rs`](../src/sql/pager/header.rs) validates all three of (magic, format version, page size) on open. A wrong magic produces `not a SQLRite database`; a wrong version or page size produces `unsupported ...` errors. The decoder accepts both v4 and v5 (anything else is rejected); the parsed `format_version` is propagated through the in-memory `DbHeader` so the writer can preserve it on resave when no version-bumping feature has been added.
 
 ## Pages 1..page_count — payload pages
 
@@ -126,14 +126,16 @@ A cell is length-prefixed; its body starts with a `kind_tag` byte:
 
 ```
 cell_length    varint          excludes itself; total bytes of kind_tag + body
-kind_tag       u8              0x01 = Local    (full row on a leaf)
-                               0x02 = Overflow (pointer to spilled body)
-                               0x03 = Interior (divider on an interior node)
-                               0x04 = Index    (one entry in an index leaf)
+kind_tag       u8              0x01 = Local         (full row on a leaf)
+                               0x02 = Overflow      (pointer to spilled body)
+                               0x03 = Interior      (divider on an interior node)
+                               0x04 = Index         (entry in a secondary-index leaf)
+                               0x05 = HNSW          (Phase 7d.3 — one HNSW node)
+                               0x06 = FTS Posting   (Phase 8c — one FTS posting list)
 body           variable        depends on kind_tag
 ```
 
-The shared prefix means `Cell::peek_rowid` works uniformly across all three kinds — useful for binary search over a page's slot directory without decoding full bodies.
+The shared prefix means `Cell::peek_rowid` works uniformly across all kinds — useful for binary search over a page's slot directory without decoding full bodies.
 
 ### Local cell body
 
@@ -205,6 +207,25 @@ value_body      variable          encoded per the Local cell's value-block rules
 
 NULL values are never indexed — `SecondaryIndex::insert` skips them — so there's no null bitmap here; a non-null value is always present.
 
+### FTS posting cell body — `KIND_FTS_POSTING` (0x06, Phase 8c)
+
+Used on the leaves of an FTS index B-Tree. Each cell carries either a posting list for one term (`term`-bytes non-empty), or — in a single sidecar cell — the per-doc length map (`term`-bytes empty). The B-Tree key is `cell_id`, a sequential integer assigned at save time; it has no meaning beyond ordering cells within their tree (so `Cell::peek_rowid`'s slot-directory ordering still works without FTS-specific page plumbing).
+
+```
+cell_id         zigzag varint     sequential B-Tree slot key (1, 2, 3, ...)
+term_len        varint            byte length of `term` (0 → sidecar cell)
+term            term_len bytes    ASCII-lowercased term per Phase 8 Q3
+count           varint            number of (rowid, value) pairs
+for each:
+  rowid         zigzag varint     the row this entry refers to
+  value         varint            term frequency for this (term, row),
+                                  or doc length when term_len == 0
+```
+
+One sidecar cell with `term_len == 0` exists per index, holding `(rowid, doc_len)` pairs for every indexed doc — including any with zero-token text — so `total_docs` and `avg_doc_len` round-trip in BM25 even on degenerate corner cases. Posting cells follow, one per unique term in lexicographic order.
+
+A single posting cell that exceeds page capacity (~4 KiB) errors at save time; overflow chaining is a Phase 8.1 stretch goal. In practice — even `'the'` in a million-row English corpus stays under the limit with the varint encoding above.
+
 ## The schema catalog: `sqlrite_master`
 
 The schema catalog is itself a table named `sqlrite_master`, stored in the same `TableLeaf` format as any user table. Its schema is hardcoded into the engine so the open path can bootstrap:
@@ -284,7 +305,8 @@ These are not all enforced on open — we validate the header strictly and rely
 - **v1** (Phases 2 / 3a / 3b) — schema catalog and table data were opaque `bincode` blobs chained across typed payload pages.
 - **v2** (Phases 3c / 3d) — cell-based storage and `sqlrite_master`. Phase 3d added interior pages without a version bump.
 - **v3** (Phase 3e) — `sqlrite_master` gains a `type` column; secondary indexes persist as their own cell-based B-Trees whose leaves carry `KIND_INDEX` cells.
-- **v4** (Phase 7a, current) — value block dispatch gains the `0x04 Vector` tag for the new `VECTOR(N)` column type. Per the [Phase 7 plan's Q8](phase-7-plan.md#q8-file-format-version-bump), later Phase 7 sub-phases (JSON storage, HNSW indexes) will add their own value/cell tags inside this same v4 envelope — no v5 mid-Phase-7. The `CREATE TABLE` SQL stored in `sqlrite_master` carries vector columns as `VECTOR(N)` in the type position; on open, the engine re-parses that SQL and reconstructs `DataType::Vector(N)` from the `Custom` AST node sqlparser produces.
+- **v4** (Phase 7a) — value block dispatch gains the `0x04 Vector` tag for the new `VECTOR(N)` column type. Per the [Phase 7 plan's Q8](phase-7-plan.md#q8-file-format-version-bump), later Phase 7 sub-phases (JSON storage, HNSW indexes) added their own value/cell tags inside this same v4 envelope. The `CREATE TABLE` SQL stored in `sqlrite_master` carries vector columns as `VECTOR(N)` in the type position; on open, the engine re-parses that SQL and reconstructs `DataType::Vector(N)` from the `Custom` AST node sqlparser produces.
+- **v5** (Phase 8c, current for FTS-bearing files) — adds the `KIND_FTS_POSTING` cell tag for persisted FTS posting lists. Bumped **on demand** per the [Phase 8 plan's Q10](phase-8-plan.md#q10-file-format-version-bump-strategy): existing v4 databases without FTS keep writing v4 across non-FTS saves; the first save with at least one FTS index attached promotes the file to v5. Decoders accept both v4 and v5; opening a v4 file with a build that supports v5 is a no-op until the user creates an FTS index.
 
 The page header (7 bytes) and chaining mechanism are stable across future phases. Phase 4's WAL introduces a sibling file (`.sqlrite-wal`) rather than changing the main file format.