feat(fts): 8f — final docs sweep for full-text search#83
Merged
Conversation
Phase 8 sub-phase 8f per docs/phase-8-plan.md. Closes the docs gap left by 8a–8e. No code changes; pure documentation. New: - docs/fts.md — canonical FTS reference. Mirrors docs/ask.md's shape: TOC, what FTS does, quickstart, SQL surface (CREATE INDEX … USING fts, fts_match, bm25_score), hybrid retrieval, tokenizer rules, BM25 parameters, lifecycle (INSERT / DELETE / UPDATE), persistence + on-demand v4→v5 bump, optimizer hook, per-surface usage (REPL / Rust / MCP / SDKs / desktop), limitations, and see-also links. Covers every Q1–Q10 design decision with cross-references back to the Phase 8 plan. Updates: - docs/supported-sql.md — new "FTS indexes (Phase 8)" subsection under CREATE INDEX, plus fts_match + bm25_score rows in the Built-in functions table, plus an "FTS query patterns" code block alongside the JSON example. - docs/architecture.md — adds src/sql/fts/ to the engine module map; updates the executor entry to mention the FTS probe shortcut (alongside HNSW); updates the on-disk storage entry to mention FTS posting lists + the on-demand v5 bump. - docs/file-format.md — bumps the "current on-disk format" line to "v4 by default, v5 on-demand"; updates the page-0 header diagram; adds KIND_FTS_POSTING (0x06) to the cell-tag table; documents the full FTS posting cell layout (cell_id, term_len, term, count, (rowid, value) pairs); adds a v5 entry to the Format evolution section. - docs/sql-engine.md — adds an "Optimizer hooks" subsection covering both try_hnsw_probe and try_fts_probe with their recognized shapes and fall-through rules. - docs/mcp.md — renames "seven tools" → "eight tools" (table, table-of-contents, mcp-inspector callout); adds a bm25_search section symmetric with vector_search. - docs/smoke-test.md — adds section 1.9b: CREATE INDEX … USING fts, fts_match WHERE filter, bm25_score top-k. - docs/_index.md — moves Phase 8 from "Future work" to its own top-level section under the docs index; updates the project- state bullet to reflect Phase 8 complete. - docs/roadmap.md — Phase 8 section moves from "active frontier draft" to "✅ complete" with a per-sub-phase ledger (8a-8f), PR numbers, and one-paragraph summaries of what each shipped. The top-of-file frontier line updates to "v0.2.0 release" as the next milestone. Total docs delta: +175 lines net across 8 files plus the new fts.md (~290 lines). Closes Phase 8. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
joaoh82
added a commit
that referenced
this pull request
May 3, 2026
First major-version bump of SQLRite. The 0.1.x cycle covered Phase 0 → Phase 7 (modernization → AI-era extensions). v0.2.0 marks Phase 8 — full-text search + hybrid retrieval — and the on-disk file-format change that comes with it. What ships in v0.2.0 ==================== Phase 8 — Full-text search + hybrid retrieval (PRs #78–#83): - 8a — Standalone FTS algorithms (tokenizer + BM25 + posting list) under src/sql/fts/, mirroring Phase 7d.1's HNSW shape. - 8b — SQL surface: CREATE INDEX … USING fts (col), fts_match(col, 'q'), bm25_score(col, 'q'), the try_fts_probe optimizer hook, and INSERT/DELETE/UPDATE lifecycle wiring (rebuild-on-save). - 8c — Cell-encoded persistence (KIND_FTS_POSTING, 0x06), stage_fts_btree / load_fts_postings, and the on-demand v4 → v5 file-format bump. Existing v4 databases without FTS keep writing v4; the first save with an FTS index attached promotes to v5. Decoders accept both. - 8d — Hybrid retrieval worked example at examples/hybrid-retrieval/ — BM25 + vector cosine via raw arithmetic (no new typed function), hand-baked vectors so it stays reproducible without a model. - 8e — sqlrite-mcp bm25_search tool, symmetric with vector_search. The MCP server now exposes 8 tools (was 7). - 8f — Final docs sweep: new docs/fts.md canonical reference; FTS sections added to supported-sql.md, architecture.md, file-format.md (KIND_FTS_POSTING layout + v5 history entry), sql-engine.md, mcp.md (count bump 7→8), smoke-test.md; _index.md and roadmap.md updated to mark Phase 8 complete. File-format change ================== This is the headline reason for the major bump: - v4 (Phase 7) — value block dispatch gained the Vector tag (Phase 7a). Phase 7's JSON, HNSW, and ask additions all lived inside the v4 envelope. - v5 (Phase 8c) — adds KIND_FTS_POSTING (0x06). On-demand: only written when the database has at least one FTS index attached. Decoders accept v4 and v5; v0.2.0 readers can open v0.1.x files without surprise. v0.1.x readers cannot open v5 files (clean "unsupported version" error). Testing ======= Engine 303 + MCP 19 + 73 across the other crates — all green at the new version. Inter-crate deps bumped from 0.1 to 0.2 in three places (Cargo.toml's optional sqlrite-ask, sqlrite-mcp's sqlrite-engine ref, sdk/wasm's sqlrite-ask ref); cargo build refreshed Cargo.lock. Manifests bumped (12 files via scripts/bump-version.sh): - Cargo.toml - sqlrite-ffi/Cargo.toml - sqlrite-ask/Cargo.toml - sqlrite-mcp/Cargo.toml - sdk/python/Cargo.toml + pyproject.toml - sdk/nodejs/Cargo.toml + package.json - sdk/wasm/Cargo.toml - desktop/src-tauri/Cargo.toml + tauri.conf.json - desktop/package.json Plus inter-crate dep refs, the Cargo.toml comment block, and sqlrite-ask/README.md's installation snippet. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sixth and final sub-phase of Phase 8. Closes the docs gap left by 8a–8e. No code changes; pure documentation.
This PR completes Phase 8 end-to-end. After it merges, the next milestone is the v0.2.0 release (the 0.1.x → 0.2.x bump marks the file-format change + new SQL surface).
What landed
New file
docs/fts.md(~290 lines) — canonical FTS reference. Mirrorsdocs/ask.md's shape: TOC, what FTS does, quickstart, SQL surface (`CREATE INDEX … USING fts`, `fts_match`, `bm25_score`), hybrid retrieval, tokenizer rules, BM25 parameters, lifecycle (INSERT / DELETE / UPDATE), persistence + on-demand v4→v5 bump, optimizer hook, per-surface usage (REPL / Rust / MCP / SDKs / desktop), limitations, and see-also links. Covers every Q1–Q10 design decision with cross-references back to the plan.Updates
docs/supported-sql.md— new "FTS indexes (Phase 8)" subsection under CREATE INDEX; `fts_match` + `bm25_score` rows added to the Built-in functions table; "FTS query patterns" code block alongside the JSON example.docs/architecture.md— adds `src/sql/fts/` to the engine module map; updates the executor entry to mention the FTS probe shortcut (alongside HNSW); updates the on-disk storage bullet to mention FTS posting lists + the on-demand v5 bump.docs/file-format.md— bumps the "current on-disk format" intro line to "v4 by default, v5 on-demand"; updates the page-0 header diagram; adds `KIND_FTS_POSTING` (0x06) to the cell-tag table; documents the full FTS posting cell layout (`cell_id`, `term_len`, `term`, `count`, `(rowid, value)` pairs); adds a v5 entry to the Format evolution section.docs/sql-engine.md— adds an "Optimizer hooks" subsection covering both `try_hnsw_probe` and `try_fts_probe` with their recognized shapes and fall-through rules.docs/mcp.md— renames "seven tools" → "eight tools" (table, TOC, mcp-inspector callout); adds a `bm25_search` section symmetric with `vector_search`.docs/smoke-test.md— new section 1.9b: `CREATE INDEX … USING fts`, `fts_match` WHERE filter, `bm25_score` top-k.docs/_index.md— Phase 8 promoted from "Future work" to its own top-level section under the docs index; project-state bullet updated to reflect Phase 8 complete.docs/roadmap.md— Phase 8 section moves from "active frontier draft" to "✅ complete" with a per-sub-phase ledger (8a–8f), PR numbers, and one-paragraph summaries. Top-of-file frontier line updates to "v0.2.0 release" as the next milestone.Test plan
After this merges
Phase 8 is shipped. Next steps:
🤖 Generated with Claude Code