Skip to content

Commit a31fb06

Browse files
joaoh82claude
joaoh82
and
claude
authored
feat(examples): SQLR-40 Node.js MCP-powered notes assistant (#141)
Adds `examples/nodejs-notes/` — a Node.js CLI that ingests a folder of markdown notes into SQLRite, then exposes the database to Claude Desktop (or any MCP client) via `sqlrite-mcp --read-only`. End user effect: `sqlrite-notes init ~/Documents/notes` + one block of JSON into Claude Desktop's config, then ask Claude about your notes. Hits two of the engine's most distinctive features end-to-end: hybrid HNSW + BM25 retrieval (the canonical fused `ORDER BY` shape from `docs/fts.md`) and the first-party MCP server. Default embedder is a deterministic offline hash bag-of-words so the demo works fully offline; `--embedder openai` switches to `text-embedding-3-small` for real semantic recall. Surfaced engine bug SQLR-8 during the `refresh` flow — HNSW panics on DELETE + INSERT within the same connection — and worked around it by splitting ingest into delete-phase → close/reopen → insert-phase. The reopen hop only runs when there are actual deletions, so first-time `init` skips it. README "Known limitations" documents it. Tests: 40 unit + integration tests via `node:test`, all passing against the prebuilt `@joaoh82/sqlrite@^0.10.0` napi-rs binding. Integration tests skip cleanly when the binding isn't installed. Cross-references updated: top-level README points at both SQLR-39 (python-agent) and SQLR-40 (nodejs-notes) under "End-to-end example apps"; `examples/README.md` gains a Node.js notes assistant section; `web/src/app/examples/page.tsx` adds a card for the new example with the right feature tags and structured-data entry. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 16a79ea commit a31fb06

28 files changed

Lines changed: 2953 additions & 5 deletions

‎README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,15 @@ Wire it into Claude Code (`~/.claude.json`):
108108

109109
`--read-only` opens the DB with a shared lock and hides the `execute` tool. Full docs + the other six tools' references in [`docs/mcp.md`](docs/mcp.md).
110110

111+
### End-to-end example apps
112+
113+
Beyond the per-language quickstarts in [`examples/`](examples/), the SQLR-38 umbrella tracks longer, opinionated example apps that exercise SQLRite in real-world shapes:
114+
115+
| App | SDK | What it shows |
116+
|---|---|---|
117+
| [Python LLM agent with persistent memory](examples/python-agent/) | Python | Vector + lexical recall, fact extraction, summaries — all in one `.sqlrite` file |
118+
| [Chat-with-your-notes via Claude Desktop MCP](examples/nodejs-notes/) | Node.js | Markdown → hybrid HNSW + BM25 index → `sqlrite-mcp --read-only` → Claude Desktop |
119+
111120
### Developer guide
112121

113122
In-depth documentation lives under [`docs/`](docs/). Start at [`docs/_index.md`](docs/_index.md) — it navigates to:

‎examples/README.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ Beyond the per-SDK quick-start tours above, the [SQLR-38 umbrella](../docs/roadm
2323
| App | Language / SDK | What it shows | Directory |
2424
|---|---|---|---|
2525
| LLM agent with persistent memory | Python | Vector + lexical recall, fact extraction, summaries — all in one `.sqlrite` file | [`python-agent/`](python-agent/) |
26+
| Chat with your notes (MCP) | Node.js | Markdown → SQLRite hybrid retrieval, served to Claude Desktop via `sqlrite-mcp --read-only` | [`nodejs-notes/`](nodejs-notes/) |
2627

2728
## Running the Rust quickstart
2829

@@ -82,6 +83,16 @@ python -m sqlrite_agent # works offline; no API key required
8283

8384
A full CLI chat agent whose long-term memory is one `.sqlrite` file. Embeds each turn, hybrid-searches over past messages and a structured `facts` table on every recall, and survives process restarts. Read [`python-agent/README.md`](python-agent/README.md) for the demo script and architecture diagram.
8485

86+
## Running the Node.js notes assistant (SQLR-40)
87+
88+
```bash
89+
cd examples/nodejs-notes
90+
npm install
91+
node bin/sqlrite-notes.mjs init ~/Documents/notes
92+
```
93+
94+
Ingests a folder of markdown notes into a `notes.sqlrite` file with HNSW + BM25 indexes, then `sqlrite-notes serve` wraps `sqlrite-mcp --read-only` so **Claude Desktop / any MCP client** can `bm25_search` / `vector_search` / `query` / `ask` your local notes directly — no cloud sync, no third-party indexer. Default embedder is fully offline (deterministic hash bag-of-words); flip to `--embedder openai` with `OPENAI_API_KEY` set for real semantic recall. Read [`nodejs-notes/README.md`](nodejs-notes/README.md) for the Claude Desktop config snippet and the hybrid-retrieval SQL walkthrough.
95+
8596
## Running the Node.js sample
8697

8798
```bash

‎examples/nodejs-notes/.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
node_modules/
2+
*.sqlrite
3+
*.sqlrite-journal
4+
.env
5+
.env.local
6+
test-fixtures-tmp/

‎examples/nodejs-notes/README.md‎

Lines changed: 344 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,344 @@
1+
# sqlrite-notes — chat with your markdown notes via Claude Desktop
2+
3+
A Node.js CLI that ingests a folder of markdown notes (Obsidian
4+
vault, Notion export, plain `~/Documents/notes`) into a SQLRite
5+
database, then exposes the database to **Claude Desktop / any MCP
6+
client** through the engine's first-party MCP server.
7+
8+
End-user effect: drop your notes folder in, paste one block into
9+
Claude Desktop's config, and ask Claude *"what did I write about
10+
CRDTs last month?"* — it answers using your local notes. No cloud
11+
sync, no third-party indexer, the entire memory is one `.sqlrite`
12+
file on disk you can open in the REPL.
13+
14+
> **Why this example?** Other "chat with your notes" demos build a
15+
> custom RAG pipeline and bolt it onto a model. This one shows that
16+
> when the database itself speaks the agent protocol, you don't need
17+
> a pipeline — *Claude drives the database directly* via
18+
> `sqlrite-mcp`. The Node.js side is just the ingest + glue.
19+
20+
## Architecture
21+
22+
```mermaid
23+
flowchart LR
24+
Notes[/"~/notes/*.md<br/>(markdown)"/] -->|sqlrite-notes init / refresh| Ingest
25+
Ingest["Ingest pipeline<br/>(walk → chunk → embed → store)"] --> DB[("notes.sqlrite<br/>documents · chunks<br/>HNSW + FTS indexes")]
26+
DB -->|"sqlrite-mcp --read-only<br/>stdio JSON-RPC"| Claude["Claude Desktop<br/>(or any MCP client)"]
27+
Claude -->|"vector_search · bm25_search · query · ask"| DB
28+
```
29+
30+
The whole stack: Node.js for the **write side** (ingest pipeline,
31+
chunking, embeddings), SQLRite for **storage + retrieval primitives**
32+
(HNSW vector index, BM25 inverted index, raw SQL), and `sqlrite-mcp`
33+
for the **read side** that Claude actually talks to. The Node CLI
34+
never touches the database while Claude is connected — that's what
35+
`--read-only` is for.
36+
37+
## Schema (v1)
38+
39+
| Table | Purpose | Indexes |
40+
|-------------|------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|
41+
| `documents` | One row per `.md` file — path, title, mtime, full body, content hash. | UNIQUE on `path`. FTS on `content` (BM25 over whole docs). |
42+
| `chunks` | One row per ~400-token slice of a document, plus a `VECTOR(384)` embedding. | HNSW on `embedding` (semantic KNN). FTS on `content` (passage BM25). |
43+
44+
Hybrid retrieval queries `chunks` and fuses BM25 + vector cosine in a
45+
single `ORDER BY` (see [`docs/fts.md`](../../docs/fts.md) for the
46+
SQL pattern; the executor's `try_fts_probe` hook serves the top-k
47+
straight from the inverted index).
48+
49+
## Install
50+
51+
The example lives inside the SQLRite monorepo for now (the umbrella
52+
ticket SQLR-38 will lift it into its own repo once we've shipped a
53+
few more).
54+
55+
```bash
56+
git clone https://github.com/joaoh82/rust_sqlite
57+
cd rust_sqlite/examples/nodejs-notes
58+
npm install
59+
```
60+
61+
`npm install` pulls **`@joaoh82/sqlrite`** (pinned to `^0.10.0`) with
62+
prebuilt napi-rs binaries for macOS-arm64, Linux x64/arm64, and
63+
Windows x64 — no Rust toolchain required for the Node side.
64+
65+
`sqlrite-mcp` is a separate Rust binary. Install it once, anywhere
66+
on your `PATH`:
67+
68+
```bash
69+
# from crates.io (~30s):
70+
cargo install sqlrite-mcp
71+
72+
# or grab a prebuilt binary from GitHub Releases:
73+
# https://github.com/joaoh82/rust_sqlite/releases
74+
```
75+
76+
If you don't want to install globally, set `SQLRITE_MCP_BIN` to its
77+
absolute path — `sqlrite-notes serve` will pick it up.
78+
79+
## Run
80+
81+
```bash
82+
# 1. Ingest a folder of markdown into a notes.sqlrite database.
83+
node bin/sqlrite-notes.mjs init ~/Documents/notes
84+
85+
# 2. Confirm it works locally — same retrieval shape Claude will see.
86+
node bin/sqlrite-notes.mjs search "what did I learn about CRDTs?"
87+
88+
# 3. Wire up Claude Desktop using the snippet printed by `init`
89+
# (also available any time via `sqlrite-notes config`).
90+
91+
# 4. Open Claude Desktop. The sqlrite-mcp tools appear in the
92+
# tool picker — `bm25_search`, `vector_search`, `query`, `ask`,
93+
# plus `list_tables` / `describe_table` / `schema_dump`.
94+
```
95+
96+
Once you've added the snippet to `claude_desktop_config.json` and
97+
restarted Claude Desktop, run a chat like:
98+
99+
> *"Summarize what I've written about Postgres over the last month."*
100+
101+
Claude will call `bm25_search` (and/or `vector_search`) against the
102+
`chunks` table, get back the matching passages, and answer with
103+
inline citations to the file path and chunk number.
104+
105+
## Zero-config: works fully offline
106+
107+
The default embedder is a **deterministic hash-based bag-of-words**
108+
embedder that runs in pure JavaScript. No API key, no network,
109+
nothing to install — `sqlrite-notes init ~/Documents/notes` works
110+
on a fresh laptop.
111+
112+
Hybrid retrieval still beats either signal alone because BM25 is
113+
already doing exact-term ranking; the hash embedder mostly carries
114+
its weight via the long-tail of co-occurring tokens.
115+
116+
For real semantic recall, switch to OpenAI:
117+
118+
```bash
119+
export OPENAI_API_KEY=sk-...
120+
node bin/sqlrite-notes.mjs init ~/Documents/notes --embedder openai
121+
```
122+
123+
Uses `text-embedding-3-small` with the `dimensions: 384` override
124+
so it matches the schema. Override the model with
125+
`--openai-model text-embedding-3-large` (and bump `--dim` if you
126+
want full-fat dimensionality).
127+
128+
## CLI surface
129+
130+
```
131+
sqlrite-notes init <dir> Ingest <dir> into the notes DB (replaces the index).
132+
sqlrite-notes refresh <dir> Re-ingest only files whose mtime/hash changed.
133+
sqlrite-notes search "<query>" Hybrid retrieval against the index (debug).
134+
sqlrite-notes serve Spawn sqlrite-mcp --read-only against the DB.
135+
sqlrite-notes stats Row counts.
136+
sqlrite-notes config Print the Claude Desktop config snippet again.
137+
```
138+
139+
Common flags:
140+
141+
```
142+
--db <path> Path to the SQLRite database file. Default: ~/.sqlrite-notes/notes.sqlrite
143+
--embedder hash|openai Embedding provider. Default: hash (offline).
144+
--dim <N> Vector dimension. Default: 384.
145+
--openai-model <name> OpenAI embedding model. Default: text-embedding-3-small.
146+
--chunk-tokens <N> Target chunk size in tokens. Default: 400.
147+
--chunk-overlap <N> Chunk overlap in tokens. Default: 60.
148+
```
149+
150+
## Claude Desktop config
151+
152+
`sqlrite-notes init` prints this block; you can also regenerate it
153+
any time with `sqlrite-notes config --db <path>`:
154+
155+
```json
156+
{
157+
"mcpServers": {
158+
"sqlrite-notes": {
159+
"command": "sqlrite-notes",
160+
"args": ["serve", "--db", "/Users/you/.sqlrite-notes/notes.sqlrite"]
161+
}
162+
}
163+
}
164+
```
165+
166+
Where it lives:
167+
168+
| Platform | Path |
169+
|---|---|
170+
| macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` |
171+
| Linux | `${XDG_CONFIG_HOME:-~/.config}/Claude/claude_desktop_config.json` |
172+
| Windows | `%APPDATA%\Claude\claude_desktop_config.json` |
173+
174+
Merge the snippet into any existing `mcpServers` block; don't
175+
overwrite the file wholesale.
176+
177+
### Try it without Claude Desktop
178+
179+
If you don't have Claude Desktop, the same MCP server works with
180+
**any** MCP client — Cursor, Codex, your own. The fastest way to
181+
verify the wiring is Anthropic's inspector:
182+
183+
```bash
184+
npx @modelcontextprotocol/inspector sqlrite-notes serve --db ./notes.sqlrite
185+
```
186+
187+
Open the URL it prints, click through the tools, type JSON args.
188+
Saves a lot of "did Claude restart correctly?" debugging.
189+
190+
## Open the DB yourself
191+
192+
The notes index is plain SQLRite — open it in the REPL whenever:
193+
194+
```bash
195+
$ cargo install sqlrite-engine # or grab a release binary
196+
$ sqlrite ./notes.sqlrite
197+
SQLRite v0.10.0
198+
sqlrite> SELECT path, title FROM documents ORDER BY mtime DESC LIMIT 5;
199+
sqlrite> SELECT path, ord, substr(content, 1, 80)
200+
...> FROM chunks
201+
...> JOIN documents ON chunks.document_id = documents.id
202+
...> WHERE fts_match(chunks.content, 'rust embedded')
203+
...> ORDER BY bm25_score(chunks.content, 'rust embedded') DESC
204+
...> LIMIT 5;
205+
```
206+
207+
This is the demo's whole point: **the notes index is just SQL**. You
208+
can query it, back it up, copy it between machines, or feed the same
209+
file into the Python / Go / WASM SDKs without converting anything.
210+
211+
## How hybrid retrieval works
212+
213+
The `search` command (and Claude, indirectly, when it composes
214+
`query` calls) runs the canonical hybrid shape from
215+
[`docs/fts.md`](../../docs/fts.md):
216+
217+
```sql
218+
SELECT id, document_id, ord, content FROM chunks
219+
WHERE fts_match(content, 'collaborative editing CRDT')
220+
ORDER BY 0.5 * bm25_score(content, 'collaborative editing CRDT')
221+
+ 0.5 * (1.0 - vec_distance_cosine(embedding, [/* query embedding */]))
222+
DESC LIMIT 5;
223+
```
224+
225+
Two things worth noting:
226+
227+
1. `vec_distance_cosine` returns a *distance* (`1 - cos(a, b)`).
228+
Hybrid scoring wants "higher is better", so we invert it.
229+
2. `fts_match` pre-filters before scoring. Paraphrases with zero
230+
shared tokens never get scored — a deliberate tradeoff. If a
231+
query produces no FTS tokens at all (e.g. a single non-ASCII
232+
word), `db.hybridSearch()` falls back to vector-only so Claude
233+
always has *something* to ground on.
234+
235+
The `-w 0..1` flag on `sqlrite-notes search` tunes the BM25 / vector
236+
balance. At `-w 1` you get pure BM25; at `-w 0` pure vector cosine.
237+
Default is 0.5.
238+
239+
## Known limitations
240+
241+
- **Concurrent `refresh` while `serve` is running.** SQLRite shipped
242+
`BEGIN CONCURRENT` writes in v0.10 (the SQLR-22 / Phase 11 work),
243+
so reading from `sqlrite-mcp --read-only` while `sqlrite-notes
244+
refresh` writes is supported in principle. *In practice we
245+
recommend stopping Claude Desktop's MCP connection during a
246+
refresh* — Claude Desktop caches the tool schemas at server-spawn
247+
time and won't notice newly-ingested files until you reload it
248+
anyway, so the gain from coexisting isn't worth the surprise.
249+
250+
- **HNSW after delete + re-insert (engine bug, SQLR-8).** The
251+
engine's HNSW chunk index panics when rows are deleted and re-
252+
inserted within the same connection lifetime — `index out of
253+
bounds` inside `DistanceMetric::compute`. The ingest pipeline
254+
works around it by splitting refresh into a delete-phase →
255+
close/reopen → insert-phase, which forces a clean index rebuild
256+
on next open. We pay the close/reopen hop only when there are
257+
actual deletions (so first-time `init` skips it). Once SQLR-8
258+
lands in the engine, `src/ingest.mjs` can drop the `db.reopen()`
259+
call.
260+
261+
- **Aggregates limited.** The engine supports `COUNT(*)`,
262+
`SUM`/`AVG`/`MIN`/`MAX` but not arbitrary expressions in `SELECT`
263+
projection beyond aggregates (see [`docs/supported-sql.md`](../../docs/supported-sql.md)).
264+
The `stats` command sticks to `COUNT(*)`.
265+
266+
- **No parameter binding yet.** Engine SDKs don't yet accept `?`
267+
placeholders ([Phase 5a.2](../../docs/roadmap.md) follow-up); all
268+
SQL strings inline literals via the `q()` helper in
269+
[`src/sqlutil.mjs`](src/sqlutil.mjs). That helper handles
270+
quoting + vector-literal encoding for the four value types we use
271+
(string / int / vector / null).
272+
273+
- **Re-ingest persistence.** `refresh` requires you to pass the
274+
source directory again (we don't yet store it inside the DB). If
275+
you forget the path, run `sqlrite-notes stats --db <path>` to
276+
confirm the index exists, then re-run `init` to rebuild from
277+
scratch — it's idempotent.
278+
279+
- **`--watch` mode** — not in v1. Re-run `refresh` manually after
280+
adding notes.
281+
282+
- **Authentication.** None. This is a single-user local tool; the
283+
MCP server inherits the spawner's filesystem privileges. Don't
284+
point it at a database you wouldn't share with whoever can read
285+
your home directory.
286+
287+
## Development
288+
289+
```bash
290+
npm install
291+
npm test # offline; runs all 40 unit + integration tests
292+
```
293+
294+
The test suite uses `node:test` and exercises:
295+
296+
- `chunker` (frontmatter, title derivation, overlap windowing)
297+
- `sqlutil` (every value type the engine accepts as a SQL literal)
298+
- `embeddings` (hash determinism + a mocked OpenAI HTTP call)
299+
- `db` (schema migration, upsert/delete/cascade, hybrid retrieval)
300+
- `ingest` (end-to-end against the markdown fixtures in
301+
[`test/fixtures/`](test/fixtures/))
302+
- `serve` + `claude-config` (binary lookup, snippet shape)
303+
304+
A few tests need the `@joaoh82/sqlrite` engine binding installed;
305+
they skip cleanly (with a message) if it isn't.
306+
307+
## Layout
308+
309+
```
310+
examples/nodejs-notes/
311+
├── package.json # @joaoh82/sqlrite pinned to ^0.10.0
312+
├── README.md # this file
313+
├── bin/
314+
│ └── sqlrite-notes.mjs # entry — calls src/cli.mjs
315+
├── src/
316+
│ ├── cli.mjs # argv parsing + command dispatch
317+
│ ├── config.mjs # defaults + path resolution
318+
│ ├── sqlutil.mjs # q() + ident() — SQL literal helpers
319+
│ ├── db.mjs # schema, migrations, every SQL string
320+
│ ├── chunker.mjs # frontmatter + heading-aware chunking
321+
│ ├── embeddings.mjs # hash + OpenAI embedders
322+
│ ├── ingest.mjs # plan → delete → reopen → insert
323+
│ ├── search.mjs # hybrid retrieval driver
324+
│ ├── serve.mjs # spawn sqlrite-mcp --read-only
325+
│ └── claude-config.mjs # Claude Desktop snippet renderer
326+
└── test/ # node --test suite
327+
├── fixtures/ # 3 short markdown notes
328+
├── chunker.test.mjs
329+
├── claude-config.test.mjs
330+
├── db.test.mjs # integration — needs @joaoh82/sqlrite
331+
├── embeddings.test.mjs
332+
├── ingest.test.mjs # integration — needs @joaoh82/sqlrite
333+
├── serve.test.mjs
334+
└── sqlutil.test.mjs
335+
```
336+
337+
The example binds only to the documented public surfaces:
338+
[`@joaoh82/sqlrite`](../../sdk/nodejs/) (`Database`, `Statement`)
339+
and [`sqlrite-mcp`](../../docs/mcp.md) as a child process. It does
340+
not reach into engine internals.
341+
342+
## License
343+
344+
MIT — same as the rest of the rust_sqlite repo.

0 commit comments

Comments
 (0)