feat(fts): 8a — standalone algorithms (tokenizer + BM25 + posting list)

[joaoh82]

Summary

First sub-phase of Phase 8 — full-text search. 8a ships the standalone algorithms only, no SQL integration yet. Mirrors the shape of Phase 7d.1's src/sql/hnsw.rs: pure algorithm, infallible API, zero project deps.

• New module src/sql/fts/ with three files
  • tokenizer.rs — ASCII split + lowercase (Q3: ASCII MVP)
  • bm25.rs — BM25+ scoring (k1=1.5, b=0.75 fixed; Q4 + Q5: no stemming, no stop list)
  • posting_list.rs — in-memory BTreeMap<term, BTreeMap<rowid, freq>> index with insert / remove / query / matches / score
• Single-line wire-up in src/sql/mod.rs (pub mod fts;)
• 22 inline unit tests (~370 LOC); whole module is ~510 LOC of code

The public surface is shaped to be directly callable from 8b without rework: PostingList::matches / score back the per-row fts_match / bm25_score scalar fns, and PostingList::query powers the bulk try_fts_probe optimizer hook.

Out of scope (later sub-phases)

Concern	Lands in
IndexMethod::Fts enum arm + executor dispatch	8b
fts_match / bm25_score scalar fn dispatch	8b
try_fts_probe optimizer hook	8b
fts_indexes: Vec<FtsIndex> on Table + INSERT/DELETE wiring	8b
KIND_FTS_POSTING cell encoding + v4→v5 file-format bump	8c
fts cargo feature gate	8e (where it gates the MCP bm25_search tool)
Hybrid retrieval worked example	8d
Docs sweep (docs/fts.md, supported-sql.md, etc.)	8f

Test plan

• cargo build --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets — clean
• cargo test sql::fts — 22 / 22 passing
• cargo test --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs — full suite green
• cargo fmt --all -- --check — no diff
• cargo clippy --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets — no new warnings on FTS code
• cargo doc --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --no-deps — module docs build

Notable test coverage

• BM25 numeric reproducibility — hand-computed reference on a 3-doc corpus, asserted within f64::EPSILON * 16. Catches any future formula drift.
• Length normalization sanity — same TF in a longer doc strictly scores lower (b > 0 working).
• IDF behavior — rare term (1 of 1000 docs) outscores common term (1000 of 1000) by ≥5×.
• 1k-doc deterministic synthetic corpus — top-10 query for a rare term surfaces exactly the 5 inserted matches, in stable order across re-runs.
• Tie-break — equal-score docs come back in ascending rowid order.

🤖 Generated with Claude Code