feat(fts): 8d — hybrid retrieval worked example (BM25 + vector)#81
Merged
Conversation
Phase 8 sub-phase 8d per docs/phase-8-plan.md. Ships a self-contained example showing how to compose Phase 8b's bm25_score with Phase 7d's vec_distance_cosine via raw arithmetic (Q8) for hybrid lexical + semantic retrieval. No new engine code — the SQL works out of the box once 8b lands. New: - examples/hybrid-retrieval/hybrid_retrieval.rs — 6-doc tech-blurb corpus with hand-baked 4-dim embeddings (no embedding-model dependency), runs three rankings on the same query: pure BM25, pure vector cosine, and 50/50 hybrid via `0.5 * bm25_score + 0.5 * (1.0 - vec_distance_cosine)`. Picks the expected docs by inspection so the example stays reproducible forever (no flaky neural-net results). - examples/hybrid-retrieval/README.md — narrative explaining when each ranking shape wins, the cosine-distance-vs-similarity gotcha, the WHERE-fts_match-pre-filter caveat, weight-tuning sketches, and a production checklist (BM25 normalization, real embeddings, cross-encoder re-rank). - Cargo.toml — registers the example; runs via `cargo run --example hybrid-retrieval`. - examples/README.md — top-level table picks up the new sample. Out of scope (later sub-phases): - MCP bm25_search tool → 8e - Docs sweep (fts.md, file-format.md, …) → 8f Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fourth sub-phase of Phase 8. Self-contained Rust example showing how to compose
bm25_score(8b) withvec_distance_cosine(Phase 7d) via raw arithmetic (Q8) for hybrid lexical + semantic retrieval.No new engine code — the SQL composition works out of the box once 8b ships. This PR is a worked example + narrative.
What landed
examples/hybrid-retrieval/hybrid_retrieval.rs— 6-doc tech-blurb corpus with hand-baked 4-dim embeddings (no embedding-model dependency). Runs three rankings on the same query: pure BM25, pure vector cosine, and 50/50 hybrid via:Vectors are picked by inspection so each query's expected ranking is obvious — the example stays reproducible forever, no flaky neural-net results.
examples/hybrid-retrieval/README.md— narrative explanation:1.0 - vec_distance_cosine(...))WHERE fts_matchpre-filter caveat (Q6 design choice)Cargo.toml— registers the example.examples/README.md— top-level table picks up the new sample alongside the Rust quickstart.Test plan
cargo run --example hybrid-retrieval— produces the expected three rankingscargo build --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets— cleancargo test --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs— 303 / 303 engine + 73 across other crates greencargo fmt --all -- --check— no diffcargo clippy --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets— no new warnings on the exampleOut of scope (later sub-phases)
bm25_searchtooldocs/fts.md,docs/file-format.md,docs/supported-sql.md, …)🤖 Generated with Claude Code