Skip to content

Commit 1832d0e

Browse files
joaoh82claude
andauthored
feat(fts): 8b — SQL surface for full-text search (#79)
Phase 8 sub-phase 8b per docs/phase-8-plan.md. Wires the standalone FTS algorithms (8a) into the SQL executor end-to-end. Mirrors the Phase 7d.2 HNSW integration shape across every touchpoint. User-visible surface: CREATE INDEX ix ON docs USING fts (body); -- predicate: row contains any query term? SELECT id FROM docs WHERE fts_match(body, 'rust embedded'); -- BM25 ranking, top-k: SELECT id FROM docs WHERE fts_match(body, 'rust embedded') ORDER BY bm25_score(body, 'rust embedded') DESC LIMIT 10; Engine plumbing: - src/sql/db/table.rs: - new FtsIndexEntry { name, column_name, index: PostingList, needs_rebuild }, mirroring HnswIndexEntry field-for-field - Table::fts_indexes alongside hnsw_indexes - maintain_fts_on_insert hook called after secondary-index + HNSW maintenance in insert_row - deep_clone propagates fts_indexes for transaction snapshots - src/sql/executor.rs: - IndexMethod::Fts arm + "fts" string-match in CREATE INDEX USING dispatch - create_fts_index validates TEXT (rejects VECTOR/JSON/INTEGER with a clear error), seeds PostingList from existing rows, pushes FtsIndexEntry - fts_match / bm25_score scalar fns in eval_function — bare column ident in arg 0, TEXT expression in arg 1; both error if no FTS index covers the column - try_fts_probe optimizer hook recognizes `ORDER BY bm25_score(col, 'q') DESC LIMIT k` and serves it from PostingList::query (top-k lookup). ASC falls through. Mirrors try_hnsw_probe's WHERE-drop posture per Q6 - DELETE / UPDATE flag fts_indexes[i].needs_rebuild = true, same shape as HNSW - name uniqueness check spans btree + hnsw + fts namespaces - src/sql/pager/mod.rs: - rebuild_dirty_fts_indexes runs at save_database start, walks current rows of dirty FTS indexes, replaces the PostingList - rebuild_fts_index replays CREATE INDEX SQL on open via execute_create_index (rootpage=0; persistence is 8c) - sqlrite_master row written for each FTS index so it survives save/reopen Twelve new integration tests in src/sql/mod.rs (covering CREATE INDEX, fts_match WHERE, bm25_score ORDER BY, incremental INSERT, DELETE/UPDATE dirty-flagging, name collisions, UNIQUE rejection, and try_fts_probe ASC fall-through), plus two persistence round-trip tests in src/sql/pager/mod.rs (re-open + query, and delete + save + reopen excludes the deleted row from FTS hits). All 287 engine tests pass; fmt + clippy clean on FTS code; no new doc warnings. Out of scope (later sub-phases): - KIND_FTS_POSTING cell encoding + v4→v5 file-format bump → 8c - Hybrid retrieval worked example → 8d - MCP bm25_search tool → 8e - Docs sweep → 8f Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 500c088 commit 1832d0e

5 files changed

Lines changed: 805 additions & 18 deletions

File tree

src/sql/db/table.rs

Lines changed: 59 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
use crate::error::{Result, SQLRiteError};
22
use crate::sql::db::secondary_index::{IndexOrigin, SecondaryIndex};
3+
use crate::sql::fts::PostingList;
34
use crate::sql::hnsw::HnswIndex;
45
use crate::sql::parser::create::CreateQuery;
56
use std::collections::{BTreeMap, HashMap};
@@ -137,6 +138,12 @@ pub struct Table {
137138
/// persisted CREATE INDEX SQL. The graph itself is NOT yet persisted —
138139
/// see Phase 7d.3 for cell-encoded graph storage.
139140
pub hnsw_indexes: Vec<HnswIndexEntry>,
141+
/// FTS inverted indexes on TEXT columns (Phase 8b). Maintained in
142+
/// lockstep with row storage on INSERT (incremental); DELETE / UPDATE
143+
/// flag `needs_rebuild` and the next save rebuilds from current rows.
144+
/// The posting lists themselves are NOT yet persisted — Phase 8c
145+
/// wires the cell-encoded `KIND_FTS_POSTING` storage.
146+
pub fts_indexes: Vec<FtsIndexEntry>,
140147
/// ROWID of most recent insert.
141148
pub last_rowid: i64,
142149
/// PRIMARY KEY column name, or "-1" if the table has no PRIMARY KEY.
@@ -163,6 +170,29 @@ pub struct HnswIndexEntry {
163170
pub needs_rebuild: bool,
164171
}
165172

173+
/// One FTS index attached to a table (Phase 8b). The inverted index
174+
/// itself is a [`PostingList`]; metadata (name, column, dirty flag)
175+
/// lives here. Mirrors [`HnswIndexEntry`] field-for-field so the
176+
/// rebuild-on-save and DELETE/UPDATE invalidation paths can use one
177+
/// pattern across both index families.
178+
#[derive(Debug, Clone)]
179+
pub struct FtsIndexEntry {
180+
/// User-supplied name from `CREATE INDEX <name> … USING fts(<col>)`.
181+
/// Unique across `secondary_indexes`, `hnsw_indexes`, and
182+
/// `fts_indexes` on a given table.
183+
pub name: String,
184+
/// The TEXT column this index covers.
185+
pub column_name: String,
186+
/// The inverted index + per-doc length cache.
187+
pub index: PostingList,
188+
/// True iff a DELETE or UPDATE-on-text-col has invalidated the
189+
/// posting lists since the last rebuild. INSERT maintains the
190+
/// index incrementally and leaves this false. The next save
191+
/// rebuilds dirty indexes from current rows before serializing
192+
/// (mirrors HNSW's Q7 strategy).
193+
pub needs_rebuild: bool,
194+
}
195+
166196
impl Table {
167197
pub fn new(create_query: CreateQuery) -> Self {
168198
let table_name = create_query.table_name;
@@ -244,6 +274,9 @@ impl Table {
244274
// time, because there's no UNIQUE-style constraint that
245275
// implies a vector index.
246276
hnsw_indexes: Vec::new(),
277+
// Same story for FTS indexes — explicit `CREATE INDEX … USING
278+
// fts(<col>)` only (Phase 8b).
279+
fts_indexes: Vec::new(),
247280
last_rowid: 0,
248281
primary_key,
249282
}
@@ -271,6 +304,8 @@ impl Table {
271304
// graph copy. Phase 4f's snapshot-rollback semantics require
272305
// the snapshot to be fully decoupled from live state.
273306
hnsw_indexes: self.hnsw_indexes.clone(),
307+
// Same fully-decoupled clone for FTS indexes (Phase 8b).
308+
fts_indexes: self.fts_indexes.clone(),
274309
last_rowid: self.last_rowid,
275310
primary_key: self.primary_key.clone(),
276311
}
@@ -908,8 +943,16 @@ impl Table {
908943
// wiring up neighbor edges, so build a get_vec closure that
909944
// pulls from the table's row storage (which we *just* updated
910945
// with the new value).
911-
if let Some(Value::Vector(new_vec)) = typed_value {
912-
self.maintain_hnsw_on_insert(key, next_rowid, &new_vec);
946+
if let Some(Value::Vector(new_vec)) = &typed_value {
947+
self.maintain_hnsw_on_insert(key, next_rowid, new_vec);
948+
}
949+
950+
// Step 4 (Phase 8b): maintain any FTS indexes on this column.
951+
// Cheap incremental update — PostingList::insert tokenizes
952+
// the value and adds postings under the new rowid. DELETE
953+
// and UPDATE take the rebuild-on-save path instead (Q7).
954+
if let Some(Value::Text(text)) = &typed_value {
955+
self.maintain_fts_on_insert(key, next_rowid, text);
913956
}
914957
}
915958
self.last_rowid = next_rowid;
@@ -948,6 +991,20 @@ impl Table {
948991
}
949992
}
950993

994+
/// After a row insert, push the new (rowid, text) into every FTS
995+
/// index whose column matches `column`. Phase 8b.
996+
///
997+
/// Mirrors [`Self::maintain_hnsw_on_insert`] but the FTS index is
998+
/// self-contained — `PostingList::insert` only needs the new doc's
999+
/// text, not the rest of the corpus, so there's no snapshot dance.
1000+
fn maintain_fts_on_insert(&mut self, column: &str, rowid: i64, text: &str) {
1001+
for entry in &mut self.fts_indexes {
1002+
if entry.column_name == column {
1003+
entry.index.insert(rowid, text);
1004+
}
1005+
}
1006+
}
1007+
9511008
/// Print the table schema to standard output in a pretty formatted way.
9521009
///
9531010
/// # Example

0 commit comments

Comments
 (0)