Skip to content

feat(fts): 8d — hybrid retrieval worked example (BM25 + vector)#81

Merged
joaoh82 merged 1 commit into
mainfrom
feat/fts-hybrid-example
May 3, 2026
Merged

feat(fts): 8d — hybrid retrieval worked example (BM25 + vector)#81
joaoh82 merged 1 commit into
mainfrom
feat/fts-hybrid-example

Conversation

@joaoh82

@joaoh82 joaoh82 commented May 3, 2026

Copy link
Copy Markdown
Owner

Summary

Fourth sub-phase of Phase 8. Self-contained Rust example showing how to compose bm25_score (8b) with vec_distance_cosine (Phase 7d) via raw arithmetic (Q8) for hybrid lexical + semantic retrieval.

No new engine code — the SQL composition works out of the box once 8b ships. This PR is a worked example + narrative.

cargo run --example hybrid-retrieval

What landed

  • examples/hybrid-retrieval/hybrid_retrieval.rs — 6-doc tech-blurb corpus with hand-baked 4-dim embeddings (no embedding-model dependency). Runs three rankings on the same query: pure BM25, pure vector cosine, and 50/50 hybrid via:

    ORDER BY 0.5 * bm25_score(body, 'q')
           + 0.5 * (1.0 - vec_distance_cosine(embedding, [...]))
    DESC LIMIT k

    Vectors are picked by inspection so each query's expected ranking is obvious — the example stays reproducible forever, no flaky neural-net results.

  • examples/hybrid-retrieval/README.md — narrative explanation:

    • When each ranking shape wins (table comparing BM25 / vector / hybrid trade-offs)
    • The cosine-distance-vs-similarity inversion gotcha (1.0 - vec_distance_cosine(...))
    • The WHERE fts_match pre-filter caveat (Q6 design choice)
    • Weight-tuning sketches (technical-doc bias, RAG bias, recency boost)
    • Production checklist (BM25 normalization, real embeddings, cross-encoder re-rank)
  • Cargo.toml — registers the example.

  • examples/README.md — top-level table picks up the new sample alongside the Rust quickstart.

Test plan

  • cargo run --example hybrid-retrieval — produces the expected three rankings
  • cargo build --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets — clean
  • cargo test --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs — 303 / 303 engine + 73 across other crates green
  • cargo fmt --all -- --check — no diff
  • cargo clippy --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets — no new warnings on the example

Out of scope (later sub-phases)

Concern Lands in
MCP bm25_search tool 8e
Docs sweep (docs/fts.md, docs/file-format.md, docs/supported-sql.md, …) 8f

🤖 Generated with Claude Code

Phase 8 sub-phase 8d per docs/phase-8-plan.md. Ships a self-contained
example showing how to compose Phase 8b's bm25_score with Phase 7d's
vec_distance_cosine via raw arithmetic (Q8) for hybrid lexical +
semantic retrieval. No new engine code — the SQL works out of the
box once 8b lands.

New:

- examples/hybrid-retrieval/hybrid_retrieval.rs — 6-doc tech-blurb
  corpus with hand-baked 4-dim embeddings (no embedding-model
  dependency), runs three rankings on the same query: pure BM25,
  pure vector cosine, and 50/50 hybrid via
  `0.5 * bm25_score + 0.5 * (1.0 - vec_distance_cosine)`. Picks the
  expected docs by inspection so the example stays reproducible
  forever (no flaky neural-net results).
- examples/hybrid-retrieval/README.md — narrative explaining when
  each ranking shape wins, the cosine-distance-vs-similarity gotcha,
  the WHERE-fts_match-pre-filter caveat, weight-tuning sketches, and
  a production checklist (BM25 normalization, real embeddings,
  cross-encoder re-rank).
- Cargo.toml — registers the example; runs via
  `cargo run --example hybrid-retrieval`.
- examples/README.md — top-level table picks up the new sample.

Out of scope (later sub-phases):
- MCP bm25_search tool                    → 8e
- Docs sweep (fts.md, file-format.md, …)  → 8f

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@joaoh82 joaoh82 merged commit 32e5b34 into main May 3, 2026
16 checks passed
@joaoh82 joaoh82 mentioned this pull request May 3, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant