docs(phase-8): plan — full-text search (BM25) + hybrid retrieval

[joaoh82]

Summary

Phase 8 planning doc, mirroring phase-7-plan.md's shape: TL;DR + sub-phases + open Q1–Q10 + implementation order + scope estimate + risks + integration map.

Awaiting your sign-off on Q1–Q10 before any code lands. Once approved, implementation kicks off at sub-phase 8a (the standalone algorithms).

What Phase 8 ships

FTS5-style inverted-index full-text search with BM25 ranking, plus hybrid retrieval (BM25 + vector-distance score fusion via raw arithmetic). The deferred 7f scope from Phase 7's Q1.

-- Build a full-text index
CREATE INDEX docs_fts ON docs USING fts(body);

-- Top-k by BM25 relevance
SELECT id FROM docs
WHERE fts_match(body, 'rust embedded database')
ORDER BY bm25_score(body, 'rust embedded database') DESC LIMIT 10;

-- Hybrid: BM25 + vector cosine, equal weights
SELECT id FROM docs
WHERE fts_match(body, 'rust embedded database')
ORDER BY 0.5 * bm25_score(body, 'rust embedded database')
       + 0.5 * (1.0 - vec_distance_cosine(embedding, [...]))
DESC LIMIT 10;

Six sub-phases

Sub-phase	Scope	Modeled after
8a algorithms (tokenizer + BM25 + posting list, no SQL)	~250 LOC	7d.1 (standalone HNSW)
8b SQL surface (CREATE INDEX … USING fts, fts_match, bm25_score, optimizer hook)	~250 LOC	7d.2 (SQL HNSW integration)
8c persistence (KIND_FTS_POSTING cell, page tree, on-demand v4→v5)	~250 LOC	7d.3 (HNSW persistence)
8d hybrid retrieval (mostly docs + worked example)	~50 LOC + ~150 docs	new — no precedent
8e MCP bm25_search tool	~50 LOC + ~100 tests	7h vector_search tool
8f docs sweep	~600 docs	follow-on to every Phase 7 release

Total: ~850 engine LOC + ~800 test + ~750 doc — comparable to Phase 7d.

Open questions for sign-off

#	Question	My recommendation
Q1	MATCH-operator syntax	A. Function-call shape fts_match(col, 'q') — no parser changes needed. SQLite-style col MATCH 'q' would require either patching sqlparser or a hand-rolled pre-parse rewrite.
Q2	Multi-column FTS — one index over many cols, or one per col?	A. Single-column per index (mirrors HNSW). Multi-column is a Phase 8.1.
Q3	Tokenizer	ASCII for MVP (split on [^A-Za-z0-9]+, lowercase). Unicode follow-up behind a unicode cargo feature.
Q4	Stemming	No. RAG queries rely on exact lexical matches.
Q5	Stop words	No. BM25's IDF naturally downweights "the", "a", etc.
Q6	Filtered FTS (fts_match(col, 'q') AND status = 'published')	A. FTS pre-filter, scalar-eval the rest. Same pattern as HNSW + WHERE.
Q7	DELETE / UPDATE on FTS-indexed tables	A. needs_rebuild + rebuild at save (mirrors HNSW).
Q8	Hybrid retrieval API	Arithmetic. No typed hybrid_score(...) function — composition via + / * is more flexible.
Q9	Add bm25_search tool to sqlrite-mcp?	Yes. Symmetric with vector_search; surfaces FTS prominently to LLM agents.
Q10	File-format version bump	On-demand. Existing v4 databases without FTS keep working as v4; first FTS-index creation rewrites page 0 to v5. Zero migration friction for current users.

Suggested release framing

The 0.1.x cycle covered Phase 0 → Phase 7 (modernization → AI-era extensions). Phase 8 changes the file format (v4 → v5 on demand) and adds a substantial new SQL surface — the natural place to bump to v0.2.0.

Alternatively fold all six sub-phases into one v0.1.26 release if the work runs small. We'll know after 8a-8c sizing is real.

Test plan

• You read the plan, push back on or approve each Q1–Q10
• Once approved, the plan doc gets updated with the resolutions in-line (same pattern Phase 7 used) and 8a kicks off as feat/fts-algorithms

Out of scope (intentionally deferred to Phase 8.1+)

• Multi-column FTS (Q2)
• Unicode tokenization (Q3) — behind a future feature flag
• Stemming + stop words (Q4 + Q5)
• Configurable BM25 k1 / b parameters
• Phrase queries (MATCH '"exact phrase"')
• Boolean query operators (AND / OR / NOT inside the query string)
• Highlight / snippet generation
• Multi-index intersection optimizer (Q6)
• Incremental DELETE / UPDATE for FTS (Q7)

🤖 Generated with Claude Code