Skip to content

feat(fts): 8b — SQL surface for full-text search#79

Merged
joaoh82 merged 1 commit into
mainfrom
feat/fts-sql-integration
May 3, 2026
Merged

feat(fts): 8b — SQL surface for full-text search#79
joaoh82 merged 1 commit into
mainfrom
feat/fts-sql-integration

Conversation

@joaoh82

@joaoh82 joaoh82 commented May 3, 2026

Copy link
Copy Markdown
Owner

Summary

Second sub-phase of Phase 8. Wires the standalone FTS algorithms shipped in #78 into the SQL executor end-to-end. Mirrors the Phase 7d.2 HNSW integration shape across every touchpoint.

User-visible:

```sql
CREATE INDEX ix ON docs USING fts (body);

-- predicate: row contains any query term?
SELECT id FROM docs WHERE fts_match(body, 'rust embedded');

-- BM25 ranking, top-k (uses the optimizer probe):
SELECT id FROM docs
WHERE fts_match(body, 'rust embedded')
ORDER BY bm25_score(body, 'rust embedded') DESC
LIMIT 10;
```

Engine plumbing

  • `Table` struct (src/sql/db/table.rs) — new `FtsIndexEntry { name, column_name, index: PostingList, needs_rebuild }` mirroring `HnswIndexEntry` field-for-field; `Table::fts_indexes` alongside `hnsw_indexes`; `maintain_fts_on_insert` hook in `insert_row` after the HNSW step.
  • Executor (src/sql/executor.rs):
    • `IndexMethod::Fts` arm + `"fts"` string-match in the `CREATE INDEX … USING` dispatch
    • `create_fts_index` validates TEXT (rejects VECTOR/JSON/INTEGER with a clear error), seeds the `PostingList` from existing rows, pushes `FtsIndexEntry`
    • `fts_match` / `bm25_score` scalar fns in `eval_function` — bare column ident in arg 0, TEXT expression in arg 1; both error if no FTS index covers the column
    • `try_fts_probe` optimizer hook recognizes `ORDER BY bm25_score(col, 'q') DESC LIMIT k` and serves it from `PostingList::query` (top-k lookup). ASC falls through to scalar eval. WHERE-drop posture mirrors `try_hnsw_probe` per Phase 8 Q6.
    • DELETE / UPDATE flag `fts_indexes[i].needs_rebuild = true`, same shape as HNSW
    • Name uniqueness check now spans btree + hnsw + fts namespaces
  • Pager (src/sql/pager/mod.rs):
    • `rebuild_dirty_fts_indexes` runs at `save_database` start, replaces the `PostingList` from current rows
    • `rebuild_fts_index` replays `CREATE INDEX … USING fts` SQL on open (rootpage=0; cell-encoded posting persistence lands in 8c)
    • Each FTS index gets a `sqlrite_master` row so it survives save/reopen

Test plan

  • 12 new integration tests in src/sql/mod.rs — CREATE INDEX, `fts_match` WHERE, `bm25_score` ORDER BY, incremental INSERT, DELETE/UPDATE dirty-flagging, name collisions, UNIQUE rejection, ASC fall-through
  • 2 persistence round-trip tests in src/sql/pager/mod.rs — re-open + query, and delete + save + reopen excludes the deleted row from FTS hits
  • `cargo build --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets` — clean
  • `cargo test --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs` — 287 / 287 engine + 73 across other crates green
  • `cargo fmt --all -- --check` — no diff
  • `cargo clippy --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --all-targets` — no new FTS warnings (engine total dropped from 22 → 19)
  • `cargo doc --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --no-deps` — no FTS doc warnings

Out of scope (later sub-phases)

Concern Lands in
`KIND_FTS_POSTING` cell encoding + v4→v5 file-format bump 8c
Hybrid retrieval worked example 8d
MCP `bm25_search` tool 8e
Docs sweep (`docs/fts.md`, `supported-sql.md`, etc.) 8f

Known limitations (documented per Phase 8 plan)

  • `try_fts_probe` ignores the WHERE clause when it fires, mirroring `try_hnsw_probe`. Canonical `WHERE fts_match(...)` is implicitly satisfied; additional WHERE conditions on the optimizer fast path are silently dropped (Q6).
  • DELETE / UPDATE flag `needs_rebuild = true` and rebuild at save (Q7); incremental delete is deferred.

🤖 Generated with Claude Code

Phase 8 sub-phase 8b per docs/phase-8-plan.md. Wires the standalone
FTS algorithms (8a) into the SQL executor end-to-end. Mirrors the
Phase 7d.2 HNSW integration shape across every touchpoint.

User-visible surface:

  CREATE INDEX ix ON docs USING fts (body);

  -- predicate: row contains any query term?
  SELECT id FROM docs WHERE fts_match(body, 'rust embedded');

  -- BM25 ranking, top-k:
  SELECT id FROM docs
   WHERE fts_match(body, 'rust embedded')
   ORDER BY bm25_score(body, 'rust embedded') DESC
   LIMIT 10;

Engine plumbing:

- src/sql/db/table.rs:
  - new FtsIndexEntry { name, column_name, index: PostingList,
    needs_rebuild }, mirroring HnswIndexEntry field-for-field
  - Table::fts_indexes alongside hnsw_indexes
  - maintain_fts_on_insert hook called after secondary-index +
    HNSW maintenance in insert_row
  - deep_clone propagates fts_indexes for transaction snapshots
- src/sql/executor.rs:
  - IndexMethod::Fts arm + "fts" string-match in CREATE INDEX
    USING dispatch
  - create_fts_index validates TEXT (rejects VECTOR/JSON/INTEGER
    with a clear error), seeds PostingList from existing rows,
    pushes FtsIndexEntry
  - fts_match / bm25_score scalar fns in eval_function — bare
    column ident in arg 0, TEXT expression in arg 1; both error if
    no FTS index covers the column
  - try_fts_probe optimizer hook recognizes
    `ORDER BY bm25_score(col, 'q') DESC LIMIT k` and serves it
    from PostingList::query (top-k lookup). ASC falls through.
    Mirrors try_hnsw_probe's WHERE-drop posture per Q6
  - DELETE / UPDATE flag fts_indexes[i].needs_rebuild = true,
    same shape as HNSW
  - name uniqueness check spans btree + hnsw + fts namespaces
- src/sql/pager/mod.rs:
  - rebuild_dirty_fts_indexes runs at save_database start, walks
    current rows of dirty FTS indexes, replaces the PostingList
  - rebuild_fts_index replays CREATE INDEX SQL on open via
    execute_create_index (rootpage=0; persistence is 8c)
  - sqlrite_master row written for each FTS index so it survives
    save/reopen

Twelve new integration tests in src/sql/mod.rs (covering CREATE
INDEX, fts_match WHERE, bm25_score ORDER BY, incremental INSERT,
DELETE/UPDATE dirty-flagging, name collisions, UNIQUE rejection,
and try_fts_probe ASC fall-through), plus two persistence
round-trip tests in src/sql/pager/mod.rs (re-open + query, and
delete + save + reopen excludes the deleted row from FTS hits).

All 287 engine tests pass; fmt + clippy clean on FTS code; no new
doc warnings.

Out of scope (later sub-phases):
- KIND_FTS_POSTING cell encoding + v4→v5 file-format bump → 8c
- Hybrid retrieval worked example                          → 8d
- MCP bm25_search tool                                     → 8e
- Docs sweep                                               → 8f

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@joaoh82 joaoh82 merged commit 1832d0e into main May 3, 2026
16 checks passed
@joaoh82 joaoh82 mentioned this pull request May 3, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant