feat(fts): 8b — SQL surface for full-text search#79
Merged
Conversation
Phase 8 sub-phase 8b per docs/phase-8-plan.md. Wires the standalone
FTS algorithms (8a) into the SQL executor end-to-end. Mirrors the
Phase 7d.2 HNSW integration shape across every touchpoint.
User-visible surface:
CREATE INDEX ix ON docs USING fts (body);
-- predicate: row contains any query term?
SELECT id FROM docs WHERE fts_match(body, 'rust embedded');
-- BM25 ranking, top-k:
SELECT id FROM docs
WHERE fts_match(body, 'rust embedded')
ORDER BY bm25_score(body, 'rust embedded') DESC
LIMIT 10;
Engine plumbing:
- src/sql/db/table.rs:
- new FtsIndexEntry { name, column_name, index: PostingList,
needs_rebuild }, mirroring HnswIndexEntry field-for-field
- Table::fts_indexes alongside hnsw_indexes
- maintain_fts_on_insert hook called after secondary-index +
HNSW maintenance in insert_row
- deep_clone propagates fts_indexes for transaction snapshots
- src/sql/executor.rs:
- IndexMethod::Fts arm + "fts" string-match in CREATE INDEX
USING dispatch
- create_fts_index validates TEXT (rejects VECTOR/JSON/INTEGER
with a clear error), seeds PostingList from existing rows,
pushes FtsIndexEntry
- fts_match / bm25_score scalar fns in eval_function — bare
column ident in arg 0, TEXT expression in arg 1; both error if
no FTS index covers the column
- try_fts_probe optimizer hook recognizes
`ORDER BY bm25_score(col, 'q') DESC LIMIT k` and serves it
from PostingList::query (top-k lookup). ASC falls through.
Mirrors try_hnsw_probe's WHERE-drop posture per Q6
- DELETE / UPDATE flag fts_indexes[i].needs_rebuild = true,
same shape as HNSW
- name uniqueness check spans btree + hnsw + fts namespaces
- src/sql/pager/mod.rs:
- rebuild_dirty_fts_indexes runs at save_database start, walks
current rows of dirty FTS indexes, replaces the PostingList
- rebuild_fts_index replays CREATE INDEX SQL on open via
execute_create_index (rootpage=0; persistence is 8c)
- sqlrite_master row written for each FTS index so it survives
save/reopen
Twelve new integration tests in src/sql/mod.rs (covering CREATE
INDEX, fts_match WHERE, bm25_score ORDER BY, incremental INSERT,
DELETE/UPDATE dirty-flagging, name collisions, UNIQUE rejection,
and try_fts_probe ASC fall-through), plus two persistence
round-trip tests in src/sql/pager/mod.rs (re-open + query, and
delete + save + reopen excludes the deleted row from FTS hits).
All 287 engine tests pass; fmt + clippy clean on FTS code; no new
doc warnings.
Out of scope (later sub-phases):
- KIND_FTS_POSTING cell encoding + v4→v5 file-format bump → 8c
- Hybrid retrieval worked example → 8d
- MCP bm25_search tool → 8e
- Docs sweep → 8f
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second sub-phase of Phase 8. Wires the standalone FTS algorithms shipped in #78 into the SQL executor end-to-end. Mirrors the Phase 7d.2 HNSW integration shape across every touchpoint.
User-visible:
```sql
CREATE INDEX ix ON docs USING fts (body);
-- predicate: row contains any query term?
SELECT id FROM docs WHERE fts_match(body, 'rust embedded');
-- BM25 ranking, top-k (uses the optimizer probe):
SELECT id FROM docs
WHERE fts_match(body, 'rust embedded')
ORDER BY bm25_score(body, 'rust embedded') DESC
LIMIT 10;
```
Engine plumbing
Test plan
Out of scope (later sub-phases)
Known limitations (documented per Phase 8 plan)
🤖 Generated with Claude Code