|
| 1 | +--- |
| 2 | +title: "Adding vector search to a SQLite-style database with HNSW" |
| 3 | +description: "How SQLRite added a VECTOR(N) column type and an HNSW index to an embedded SQL engine — distance functions, the executor shortcut, and the design tradeoffs of putting an ANN graph next to a B-tree." |
| 4 | +publishedAt: "2026-04-22" |
| 5 | +author: "Joao Henrique Machado Silva" |
| 6 | +tags: ["sqlrite", "vector-search", "hnsw", "rust", "rag", "ai"] |
| 7 | +primaryKeyword: "embedded vector search HNSW Rust" |
| 8 | +--- |
| 9 | + |
| 10 | +Vector search inside an embedded SQL engine sounds like it should be |
| 11 | +weird, and it sort of is. SQL was built for sets and predicates. ANN |
| 12 | +search is geometry. The two languages don't quite agree on what a |
| 13 | +"row" is, what an "index" does, or what `ORDER BY` should mean when |
| 14 | +the order is "closer to this 384-dimensional point." |
| 15 | + |
| 16 | +But for the AI-shaped year we're in, this is the integration users |
| 17 | +actually want. RAG over local notes. Agent memory. Hybrid retrieval |
| 18 | +that combines a BM25 score with a cosine distance. Today those use |
| 19 | +cases get stitched together with `sqlite-vec` plus FTS5 plus glue |
| 20 | +code; it works, but it's "you, the integrator, are responsible." For |
| 21 | +[SQLRite](https://github.com/joaoh82/rust_sqlite) Phase 7d, I wanted |
| 22 | +the integration to be the engine's job. (For the philosophy behind |
| 23 | +that decision, see [the origin-story post](/blog/why-im-building-sqlrite). |
| 24 | +For how rows actually get to disk underneath all this, see |
| 25 | +[the storage deep-dive](/blog/how-sqlrite-stores-rows-on-disk).) |
| 26 | + |
| 27 | +This post is a retrospective on how that landed: what the API looks |
| 28 | +like, what the index does under the hood, and which tradeoffs were |
| 29 | +not obvious until I'd shipped the wrong version of them. |
| 30 | + |
| 31 | +## The user-facing API |
| 32 | + |
| 33 | +Here is the whole thing: |
| 34 | + |
| 35 | +```sql |
| 36 | +CREATE TABLE docs ( |
| 37 | + id INTEGER PRIMARY KEY, |
| 38 | + body TEXT, |
| 39 | + embedding VECTOR(384) |
| 40 | +); |
| 41 | + |
| 42 | +CREATE INDEX docs_emb ON docs(embedding) USING HNSW; |
| 43 | + |
| 44 | +INSERT INTO docs (body, embedding) VALUES |
| 45 | + ('rust embedded database', [0.13, -0.02, ...]); |
| 46 | + |
| 47 | +SELECT id, vec_distance_cosine(embedding, ?) AS dist |
| 48 | +FROM docs |
| 49 | +ORDER BY dist ASC |
| 50 | +LIMIT 10; |
| 51 | +``` |
| 52 | + |
| 53 | +`VECTOR(N)` is a column type. The dimension is fixed per column; rows |
| 54 | +that don't conform get rejected with a typed error. Three distance |
| 55 | +functions ship: `vec_distance_cosine`, `vec_distance_dot`, |
| 56 | +`vec_distance_l2`. There's a single `USING HNSW` index type. That's |
| 57 | +the surface. |
| 58 | + |
| 59 | +Bracket array literals are first-class — `[0.13, -0.02, ...]` is a |
| 60 | +vector value, not a string. From Rust: |
| 61 | + |
| 62 | +```rust |
| 63 | +use sqlrite::{Connection, Value}; |
| 64 | + |
| 65 | +let mut conn = Connection::open("memory.sqlrite")?; |
| 66 | +let mut stmt = conn.prepare_cached( |
| 67 | + "SELECT id FROM docs ORDER BY vec_distance_cosine(embedding, ?) ASC LIMIT 10", |
| 68 | +)?; |
| 69 | +let rows = stmt.query_with_params(&[Value::Vector(query_vec)])?.collect_all()?; |
| 70 | +``` |
| 71 | + |
| 72 | +A `Value::Vector(Vec<f32>)` binds where a literal would otherwise go, |
| 73 | +which means *prepared* k-NN queries still take the index shortcut. |
| 74 | +That last bit was its own little design problem; more on it below. |
| 75 | + |
| 76 | +## The HNSW index |
| 77 | + |
| 78 | +[HNSW](https://arxiv.org/abs/1603.09320) (Hierarchical Navigable |
| 79 | +Small World) is the standard graph-based ANN index. The summary, in |
| 80 | +two sentences: every vector is a node in a multi-layer graph where |
| 81 | +higher layers are sparser; you start a query at the top, greedily |
| 82 | +hop toward the query, and descend layers as you get closer. It's not |
| 83 | +the only ANN index in the world — IVF-PQ, ScaNN, DiskANN, and |
| 84 | +FAISS-style flat-with-quantization all have their use cases — but |
| 85 | +HNSW has two properties that matter for embedded: |
| 86 | + |
| 87 | +- **Sub-linear search time** without giving up much recall. |
| 88 | +- **No training step.** You insert, you search. There is no "first |
| 89 | + fit a centroid model on a representative batch." For an embedded |
| 90 | + database that doesn't know what the workload looks like before the |
| 91 | + user installs it, this is exactly the right tradeoff. |
| 92 | + |
| 93 | +The downside is build cost. Inserting a vector into HNSW does a |
| 94 | +short-radius search to find its neighbors, then wires them together. |
| 95 | +That's a O(log n) walk per insert with a fan-out of `M` (the |
| 96 | +neighbor cap, default 16). On a million-row dataset, that adds up. |
| 97 | +SQLRite's first version of insert was naïve and bench-bound; the |
| 98 | +fixed version batches insertions and reuses search state. |
| 99 | + |
| 100 | +## Where the graph lives |
| 101 | + |
| 102 | +Two options: in the SQLite-style file format, or in memory rebuilt |
| 103 | +on open. |
| 104 | + |
| 105 | +I tried both. The persistent option is, eventually, the right answer — |
| 106 | +but it's a much bigger change than it sounds. HNSW graphs aren't |
| 107 | +B-trees; their access pattern is "follow many small pointers across |
| 108 | +the graph," which is what virtual memory and `mmap` are good at and |
| 109 | +what 4 KiB pages are not. Persisting the graph in 4 KiB pages either |
| 110 | +spreads neighbors across many pages (fine for spinning disks, awful |
| 111 | +for cache locality) or packs neighbors next to nodes (which means |
| 112 | +node insertions can reshape the layout, which means the diff-based |
| 113 | +pager writes a lot more pages per commit, which is exactly what we |
| 114 | +spent the last phase removing). |
| 115 | + |
| 116 | +So Phase 7d ships **HNSW in memory, rebuilt on open**. The vectors |
| 117 | +themselves still live in the table's B-tree — they're just typed |
| 118 | +column values — so durability and crash safety are unchanged. On |
| 119 | +`Connection::open`, the engine walks the table once and re-inserts |
| 120 | +each vector into a fresh HNSW. For a million 384-dimensional rows, |
| 121 | +this takes a few seconds. For the embedded use cases I care about |
| 122 | +right now (notes, chat history, codebases at human scale), that's |
| 123 | +fine. For bigger tables, it isn't, and persistence is a Phase 9 |
| 124 | +problem. |
| 125 | + |
| 126 | +The thing I want to defend here: **the wrong persistence is worse |
| 127 | +than no persistence.** A reader who opens a million-vector database, |
| 128 | +waits 800 ms for the graph rebuild, and runs a query that returns in |
| 129 | +1.5 ms is having a fine time. A reader who opens the same database, |
| 130 | +loads a 200 MB graph file from disk that doesn't quite match the |
| 131 | +table any more, and gets a stale answer is *not*. Embedded |
| 132 | +databases live or die on the user's trust that what they query is |
| 133 | +what they wrote. |
| 134 | + |
| 135 | +## Wiring it into the executor |
| 136 | + |
| 137 | +This is where SQL stops being convenient. The query |
| 138 | + |
| 139 | +```sql |
| 140 | +SELECT id, vec_distance_cosine(embedding, ?) AS dist |
| 141 | +FROM docs |
| 142 | +ORDER BY dist ASC |
| 143 | +LIMIT 10; |
| 144 | +``` |
| 145 | + |
| 146 | +…parses as a normal scan-then-sort plan. A naïve executor would |
| 147 | +compute `vec_distance_cosine` for every row in `docs`, sort, and |
| 148 | +take the top 10. That's O(n) per query. We have an HNSW index sitting |
| 149 | +right there. |
| 150 | + |
| 151 | +The executor recognizes a specific shape — `ORDER BY` a |
| 152 | +`vec_distance_*` over the indexed column, `LIMIT k`, no other |
| 153 | +side effects — and rewrites it into an HNSW probe with a bounded |
| 154 | +top-k heap. If the shape doesn't match (because of a `WHERE` clause |
| 155 | +that filters on a non-vector column, say), the query falls back to a |
| 156 | +full scan and the optimizer flags it. We also keep the `WHERE` |
| 157 | +predicate path correct under the rewrite — the heap accepts a row |
| 158 | +only if it satisfies the surviving predicates. This is the |
| 159 | +"post-filter" pattern; it's the same shape Pinecone and pgvector |
| 160 | +use. |
| 161 | + |
| 162 | +There's a third path the executor takes: when `LIMIT k` is large |
| 163 | +enough relative to `n`, the rewrite isn't worth it (HNSW with |
| 164 | +`ef_search` tuned to recall everything stops being sub-linear), and |
| 165 | +we just do the full scan. The threshold is currently a constant; it |
| 166 | +should eventually live in a per-index PRAGMA. |
| 167 | + |
| 168 | +## The thing nobody tells you about prepared statements |
| 169 | + |
| 170 | +A prepared statement looks like a tiny optimization — parse the SQL |
| 171 | +once, run the plan many times. For vector search, it is also a |
| 172 | +correctness requirement: if `?` binds to a different vector each |
| 173 | +time, the plan must still take the HNSW shortcut. |
| 174 | + |
| 175 | +The naïve implementation parses the placeholder as "an opaque |
| 176 | +parameter," and the optimizer can't tell at plan time whether the |
| 177 | +parameter is a vector or a string or an integer. So it can't safely |
| 178 | +rewrite the plan. So you fall back to the full scan, every time. |
| 179 | + |
| 180 | +The fix is small but not obvious: the **prepared cache stores plans |
| 181 | +keyed by the AST shape with parameter slots typed.** When the AST is |
| 182 | +of the form `ORDER BY vec_distance_*(col, ?) LIMIT k` and `col` has |
| 183 | +an HNSW index, we install the rewritten plan and remember that the |
| 184 | +parameter must bind to a `Value::Vector(_)`. At bind time we |
| 185 | +type-check; if the user binds a `Value::Text(_)` instead, we error. |
| 186 | +This is the same trick a real query optimizer uses for parameter |
| 187 | +sniffing, dressed up in 30 lines of code. |
| 188 | + |
| 189 | +`prepare_cached` keeps a per-connection LRU plan cache (default cap |
| 190 | +16). On a hot RAG path, the cache hit means the SQL never parses |
| 191 | +twice and the plan never gets rewritten twice. The cost is exactly |
| 192 | +the lookup. |
| 193 | + |
| 194 | +## A short list of things that surprised me |
| 195 | + |
| 196 | +- **Distance choice matters.** Cosine, dot product, and L2 are not |
| 197 | + interchangeable, and embeddings shipped by different model families |
| 198 | + expect different ones. We support all three, but the README needed |
| 199 | + a paragraph telling users to pick the one their model recommends. |
| 200 | +- **Vectors are huge.** A `VECTOR(1024)` is 4 KiB by itself, which |
| 201 | + means it always overflows. That's fine, but it changes the cost |
| 202 | + curve of inserts in a table that has *only* vector columns versus |
| 203 | + one that mixes vectors with small typed columns. |
| 204 | +- **`LIMIT 1` is the canary.** ANN indexes shine when `k` is small. |
| 205 | + If your benchmarks all use `LIMIT 100`, you'll think HNSW is |
| 206 | + modestly faster than scan. If your benchmarks use `LIMIT 10`, |
| 207 | + HNSW dunks on it. RAG retrieval is in the second regime. |
| 208 | +- **Recall is a knob, not a constant.** HNSW has an `ef_search` |
| 209 | + parameter that trades recall for latency. Phase 7d picked a |
| 210 | + conservative default; an upcoming PRAGMA exposes it. |
| 211 | + |
| 212 | +## Where this leaves us |
| 213 | + |
| 214 | +Vector search as a first-class feature of the embedded SQL engine, |
| 215 | +with prepared statements that take the shortcut, with a graph that |
| 216 | +rebuilds on open and otherwise feels invisible, with three distance |
| 217 | +functions and one index type. It is the integration I wanted the |
| 218 | +engine to provide, and it's the bedrock for the next phase: BM25 |
| 219 | +full-text search and **hybrid retrieval**, where `bm25_score` and |
| 220 | +`vec_distance_cosine` get to compose in the same query. That's |
| 221 | +[Phase 8](https://github.com/joaoh82/rust_sqlite/blob/main/docs/roadmap.md), |
| 222 | +in flight now. |
| 223 | + |
| 224 | +If you want to play with it, there's a runnable example in |
| 225 | +[`examples/vector-search/`](https://github.com/joaoh82/rust_sqlite/tree/main/examples) |
| 226 | +of the repo, and the [docs page](/docs) covers the |
| 227 | +[query patterns](/docs#vector). The MCP server's `vector_search` |
| 228 | +tool exposes the same API to AI clients with two lines of config — |
| 229 | +the smallest "agent memory" demo I've seen, with no extra moving |
| 230 | +parts. |
| 231 | + |
| 232 | +If SQLRite is useful to you, ⭐ the |
| 233 | +[GitHub repo](https://github.com/joaoh82/rust_sqlite) — visibility |
| 234 | +is the bottleneck for projects like this, and a single star helps |
| 235 | +more than you'd think. |
0 commit comments