Commit 1832d0e
feat(fts): 8b — SQL surface for full-text search (#79)
Phase 8 sub-phase 8b per docs/phase-8-plan.md. Wires the standalone
FTS algorithms (8a) into the SQL executor end-to-end. Mirrors the
Phase 7d.2 HNSW integration shape across every touchpoint.
User-visible surface:
CREATE INDEX ix ON docs USING fts (body);
-- predicate: row contains any query term?
SELECT id FROM docs WHERE fts_match(body, 'rust embedded');
-- BM25 ranking, top-k:
SELECT id FROM docs
WHERE fts_match(body, 'rust embedded')
ORDER BY bm25_score(body, 'rust embedded') DESC
LIMIT 10;
Engine plumbing:
- src/sql/db/table.rs:
- new FtsIndexEntry { name, column_name, index: PostingList,
needs_rebuild }, mirroring HnswIndexEntry field-for-field
- Table::fts_indexes alongside hnsw_indexes
- maintain_fts_on_insert hook called after secondary-index +
HNSW maintenance in insert_row
- deep_clone propagates fts_indexes for transaction snapshots
- src/sql/executor.rs:
- IndexMethod::Fts arm + "fts" string-match in CREATE INDEX
USING dispatch
- create_fts_index validates TEXT (rejects VECTOR/JSON/INTEGER
with a clear error), seeds PostingList from existing rows,
pushes FtsIndexEntry
- fts_match / bm25_score scalar fns in eval_function — bare
column ident in arg 0, TEXT expression in arg 1; both error if
no FTS index covers the column
- try_fts_probe optimizer hook recognizes
`ORDER BY bm25_score(col, 'q') DESC LIMIT k` and serves it
from PostingList::query (top-k lookup). ASC falls through.
Mirrors try_hnsw_probe's WHERE-drop posture per Q6
- DELETE / UPDATE flag fts_indexes[i].needs_rebuild = true,
same shape as HNSW
- name uniqueness check spans btree + hnsw + fts namespaces
- src/sql/pager/mod.rs:
- rebuild_dirty_fts_indexes runs at save_database start, walks
current rows of dirty FTS indexes, replaces the PostingList
- rebuild_fts_index replays CREATE INDEX SQL on open via
execute_create_index (rootpage=0; persistence is 8c)
- sqlrite_master row written for each FTS index so it survives
save/reopen
Twelve new integration tests in src/sql/mod.rs (covering CREATE
INDEX, fts_match WHERE, bm25_score ORDER BY, incremental INSERT,
DELETE/UPDATE dirty-flagging, name collisions, UNIQUE rejection,
and try_fts_probe ASC fall-through), plus two persistence
round-trip tests in src/sql/pager/mod.rs (re-open + query, and
delete + save + reopen excludes the deleted row from FTS hits).
All 287 engine tests pass; fmt + clippy clean on FTS code; no new
doc warnings.
Out of scope (later sub-phases):
- KIND_FTS_POSTING cell encoding + v4→v5 file-format bump → 8c
- Hybrid retrieval worked example → 8d
- MCP bm25_search tool → 8e
- Docs sweep → 8f
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 500c088 commit 1832d0e
5 files changed
Lines changed: 805 additions & 18 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| |||
137 | 138 | | |
138 | 139 | | |
139 | 140 | | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
140 | 147 | | |
141 | 148 | | |
142 | 149 | | |
| |||
163 | 170 | | |
164 | 171 | | |
165 | 172 | | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
166 | 196 | | |
167 | 197 | | |
168 | 198 | | |
| |||
244 | 274 | | |
245 | 275 | | |
246 | 276 | | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
247 | 280 | | |
248 | 281 | | |
249 | 282 | | |
| |||
271 | 304 | | |
272 | 305 | | |
273 | 306 | | |
| 307 | + | |
| 308 | + | |
274 | 309 | | |
275 | 310 | | |
276 | 311 | | |
| |||
908 | 943 | | |
909 | 944 | | |
910 | 945 | | |
911 | | - | |
912 | | - | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
913 | 956 | | |
914 | 957 | | |
915 | 958 | | |
| |||
948 | 991 | | |
949 | 992 | | |
950 | 993 | | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
951 | 1008 | | |
952 | 1009 | | |
953 | 1010 | | |
| |||
0 commit comments