Skip to content
This repository was archived by the owner on Feb 18, 2026. It is now read-only.

Commit 5a766ee

Browse files
mceachenmceachen
committed
feat(types): add duck-typed DatabaseLike interface for multi-library support
Replace hard dependency on @photostructure/sqlite with DatabaseLike/StatementLike interfaces that accept any SQLite library (node:sqlite, better-sqlite3, @photostructure/sqlite, or custom wrappers). - Strip file extension in loadDiskAnnExtension for cross-library compat - Add test factory to run tests against all available implementations - Add docs/compatibility.md with migration guide and interface docs
1 parent 2eb4cee commit 5a766ee

9 files changed

Lines changed: 1660 additions & 384 deletions

File tree

‎README.md‎

Lines changed: 71 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,19 +18,50 @@ A standalone SQLite extension implementing the [DiskANN algorithm](https://githu
1818
- Incremental insert/delete support
1919
- Cross-platform: Linux, macOS, Windows (x64, arm64)
2020

21+
## Database Compatibility
22+
23+
This package works with multiple SQLite library implementations through duck typing:
24+
25+
| Library | Availability | Notes |
26+
| -------------------------- | ---------------------- | ---------------------------------------- |
27+
| **@photostructure/sqlite** | npm package | ✅ Stable, 100% `node:sqlite` compatible |
28+
| **better-sqlite3** | npm package | ✅ Mature, stable, widely used |
29+
| **node:sqlite** | Node.js 22.5+ built-in | ⚠️ Experimental (requires flag) |
30+
31+
### Which should I use?
32+
33+
- **Production**: Use `better-sqlite3` or `@photostructure/sqlite` (both stable)
34+
- **Node 22.5+**: Can use built-in `node:sqlite` (zero dependencies, but experimental)
35+
- **Existing projects**: Continue using whatever you already have
36+
2137
## Installation
2238

2339
```bash
40+
# Install sqlite-diskann
2441
npm install @photostructure/sqlite-diskann
42+
43+
# Install a SQLite library (choose one):
44+
45+
# Option 1: @photostructure/sqlite (recommended for production)
46+
npm install @photostructure/sqlite
47+
48+
# Option 2: better-sqlite3 (recommended for production)
49+
npm install better-sqlite3
50+
51+
# Option 3: Use Node.js 22.5+ built-in (no install needed)
52+
# Requires Node.js >= 22.5.0 and --experimental-sqlite flag
53+
# ⚠️ Still experimental (requires --experimental-sqlite flag)
2554
```
2655

2756
## Quick Start
2857

58+
### With @photostructure/sqlite
59+
2960
```typescript
30-
import Database from "@photostructure/sqlite";
61+
import { DatabaseSync } from "@photostructure/sqlite";
3162
import { loadDiskAnnExtension } from "@photostructure/sqlite-diskann";
3263

33-
const db = new Database(":memory:");
64+
const db = new DatabaseSync(":memory:", { allowExtension: true });
3465
loadDiskAnnExtension(db);
3566

3667
// Create index for 128-dimensional vectors
@@ -53,6 +84,42 @@ const results = db
5384
.all(vector);
5485
```
5586

87+
### With better-sqlite3
88+
89+
```typescript
90+
import Database from "better-sqlite3";
91+
import { loadDiskAnnExtension } from "@photostructure/sqlite-diskann";
92+
93+
const db = new Database(":memory:");
94+
loadDiskAnnExtension(db);
95+
96+
// Now you can use DiskANN functions
97+
db.exec(`
98+
CREATE VIRTUAL TABLE embeddings USING diskann(
99+
dimension=512,
100+
metric=cosine
101+
)
102+
`);
103+
```
104+
105+
### With node:sqlite (Node 22.5+, experimental)
106+
107+
```typescript
108+
import { DatabaseSync } from "node:sqlite";
109+
import { loadDiskAnnExtension } from "@photostructure/sqlite-diskann";
110+
111+
const db = new DatabaseSync(":memory:", { allowExtension: true });
112+
loadDiskAnnExtension(db);
113+
114+
// Now you can use DiskANN functions
115+
db.exec(`
116+
CREATE VIRTUAL TABLE embeddings USING diskann(
117+
dimension=512,
118+
metric=cosine
119+
)
120+
`);
121+
```
122+
56123
## Why DiskANN?
57124

58125
Most SQLite vector extensions either:
@@ -104,7 +171,8 @@ sudo apt-get install build-essential clang-tidy valgrind
104171
make all
105172
106173
# Test
107-
make test # C unit tests
174+
make test # C unit tests (126 tests)
175+
make test-stress # Stress tests (300k/100k vectors, ~30 min)
108176
make asan # AddressSanitizer
109177
make valgrind # Memory leak detection
110178
npm test # TypeScript tests

‎docs/compatibility.md‎

Lines changed: 265 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,265 @@
1+
# SQLite Implementation Compatibility
2+
3+
## Overview
4+
5+
sqlite-diskann uses **duck typing** to support multiple SQLite library implementations. Any library providing the minimal required interface will work automatically, without needing to be explicitly listed as a dependency.
6+
7+
## Required Interface
8+
9+
Your SQLite library must provide these methods:
10+
11+
```typescript
12+
interface DatabaseLike {
13+
loadExtension(path: string, entryPoint?: string): void;
14+
exec(sql: string): void;
15+
prepare(sql: string): StatementLike;
16+
}
17+
18+
interface StatementLike {
19+
run(...params: any[]): { changes: number; lastInsertRowid: number | bigint };
20+
all(...params: any[]): any[];
21+
}
22+
```
23+
24+
This is the **complete** interface required - just 3 database methods and 2 statement methods.
25+
26+
## Tested Implementations
27+
28+
### node:sqlite (Node 22.5+, experimental)
29+
30+
- **Module**: Built-in to Node.js (since 22.5.0)
31+
- **Class**: `DatabaseSync`
32+
- **Compatibility**: ✅ 100% compatible
33+
- **Performance**: Fastest (no C++ addon loading overhead)
34+
- **Installation**: None (built-in)
35+
- **Status**: ⚠️ **Experimental** (stability level 1.1 - Active development)
36+
37+
```typescript
38+
import { DatabaseSync } from "node:sqlite";
39+
import { loadDiskAnnExtension } from "@photostructure/sqlite-diskann";
40+
41+
const db = new DatabaseSync(":memory:", { allowExtension: true });
42+
loadDiskAnnExtension(db);
43+
```
44+
45+
**Requirements**:
46+
47+
- Node.js >= 22.5.0
48+
- `--experimental-sqlite` flag required
49+
- Still experimental as of this writing (not recommended for production)
50+
51+
**Run with:**
52+
53+
```bash
54+
node --experimental-sqlite your-script.js
55+
```
56+
57+
### better-sqlite3
58+
59+
- **Module**: `better-sqlite3` npm package
60+
- **Class**: `Database`
61+
- **Compatibility**: ✅ 100% compatible
62+
- **Performance**: Excellent
63+
- **Installation**: `npm install better-sqlite3`
64+
65+
```typescript
66+
import Database from "better-sqlite3";
67+
import { loadDiskAnnExtension } from "@photostructure/sqlite-diskann";
68+
69+
const db = new Database(":memory:");
70+
loadDiskAnnExtension(db);
71+
```
72+
73+
### @photostructure/sqlite
74+
75+
- **Module**: `@photostructure/sqlite` npm package
76+
- **Class**: `DatabaseSync`
77+
- **Compatibility**: ✅ 100% compatible
78+
- **Performance**: Identical to node:sqlite
79+
- **Installation**: `npm install @photostructure/sqlite`
80+
81+
```typescript
82+
import { DatabaseSync } from "@photostructure/sqlite";
83+
import { loadDiskAnnExtension } from "@photostructure/sqlite-diskann";
84+
85+
const db = new DatabaseSync(":memory:", { allowExtension: true });
86+
loadDiskAnnExtension(db);
87+
```
88+
89+
**Note**: This is an independent implementation, written from scratch, that is strictly API-compatible with the latest version of Node.js's `node:sqlite` module. It supports older Node.js versions that lack the built-in `node:sqlite` module.
90+
91+
## Known Differences Between Implementations
92+
93+
All three implementations are compatible, but there are minor differences to be aware of:
94+
95+
### Return Type: `lastInsertRowid`
96+
97+
All three implementations return `{ changes: number, lastInsertRowid: number | bigint }` from `stmt.run()`. The `lastInsertRowid` field can be either `number` or `bigint` depending on the row ID value.
98+
99+
**Recommended handling:**
100+
101+
```typescript
102+
const { lastInsertRowid } = stmt.run(params);
103+
const id =
104+
typeof lastInsertRowid === "bigint" ? Number(lastInsertRowid) : lastInsertRowid;
105+
```
106+
107+
### Error Messages
108+
109+
Each implementation throws different error types and messages. Your error handling should check message content rather than error type:
110+
111+
```typescript
112+
try {
113+
loadDiskAnnExtension(db);
114+
} catch (error) {
115+
console.error("Failed to load extension:", error.message);
116+
}
117+
```
118+
119+
### Extension Loading
120+
121+
- **node:sqlite**: Requires `--experimental-sqlite` flag (all versions, still experimental)
122+
- **better-sqlite3**: No flags needed, stable
123+
- **@photostructure/sqlite**: No flags needed, stable
124+
125+
## Using Your Own SQLite Wrapper
126+
127+
If you have a custom SQLite wrapper, it will work with sqlite-diskann if it provides the required interface:
128+
129+
```typescript
130+
class MyCustomDatabase {
131+
loadExtension(path: string, entryPoint?: string): void {
132+
// Your implementation
133+
}
134+
135+
exec(sql: string): void {
136+
// Your implementation
137+
}
138+
139+
prepare(sql: string): MyCustomStatement {
140+
// Your implementation
141+
return new MyCustomStatement(sql);
142+
}
143+
}
144+
145+
class MyCustomStatement {
146+
run(...params: any[]): { changes: number; lastInsertRowid: number | bigint } {
147+
// Your implementation
148+
return { changes: 1, lastInsertRowid: 123 };
149+
}
150+
151+
all(...params: any[]): any[] {
152+
// Your implementation
153+
return [];
154+
}
155+
}
156+
```
157+
158+
Then use it with sqlite-diskann:
159+
160+
```typescript
161+
import { loadDiskAnnExtension } from "@photostructure/sqlite-diskann";
162+
163+
const db = new MyCustomDatabase();
164+
loadDiskAnnExtension(db); // Works!
165+
```
166+
167+
## Type Safety
168+
169+
TypeScript will accept any object with the correct shape:
170+
171+
```typescript
172+
// This type-checks but will fail at runtime
173+
const fakeDb = {
174+
exec: () => {},
175+
prepare: () => ({ run: () => ({}), all: () => [] }),
176+
loadExtension: () => {
177+
throw new Error("Not implemented");
178+
},
179+
};
180+
181+
loadDiskAnnExtension(fakeDb); // Compiles, throws at runtime
182+
```
183+
184+
This is intentional - duck typing trades compile-time safety for runtime flexibility. The benefit is that **any** SQLite library works without modification.
185+
186+
## Migration Guide
187+
188+
### Switching Between Implementations
189+
190+
All three implementations have identical APIs for the methods we use. To switch:
191+
192+
#### From @photostructure/sqlite to node:sqlite (Node 22+)
193+
194+
```bash
195+
npm uninstall @photostructure/sqlite
196+
```
197+
198+
```diff
199+
-import { DatabaseSync } from "@photostructure/sqlite";
200+
+import { DatabaseSync } from "node:sqlite";
201+
202+
const db = new DatabaseSync(":memory:", { allowExtension: true });
203+
// Rest of code unchanged
204+
```
205+
206+
#### From @photostructure/sqlite to better-sqlite3
207+
208+
```bash
209+
npm uninstall @photostructure/sqlite
210+
npm install better-sqlite3
211+
```
212+
213+
```diff
214+
-import { DatabaseSync } from "@photostructure/sqlite";
215+
+import Database from "better-sqlite3";
216+
217+
-const db = new DatabaseSync(":memory:", { allowExtension: true });
218+
+const db = new Database(":memory:");
219+
// Rest of code unchanged
220+
```
221+
222+
## Advantages of Duck Typing
223+
224+
### No Peer Dependency Warnings
225+
226+
Without peer dependencies, users never see warnings like:
227+
228+
```
229+
npm WARN sqlite-diskann@1.0.0 requires a peer of better-sqlite3@>=11.0.0
230+
```
231+
232+
### Zero Dependencies
233+
234+
Users with Node 22+ don't need to install any additional packages. The package has **zero runtime dependencies**.
235+
236+
### Future-Proof
237+
238+
If a new SQLite library is released tomorrow with the same interface, it will work with sqlite-diskann without any changes.
239+
240+
### Flexibility
241+
242+
Users can:
243+
244+
- Mock the database for testing
245+
- Use wrapped/proxied database instances
246+
- Switch implementations without changing application code
247+
- Use internal/enterprise SQLite libraries
248+
249+
## Testing
250+
251+
The test suite runs against all available implementations automatically:
252+
253+
```bash
254+
npm run test:ts
255+
```
256+
257+
Each available implementation gets its own test suite. If an implementation is not installed or not available on the current Node version (e.g., `node:sqlite` on Node 20), those tests are automatically skipped.
258+
259+
## Summary
260+
261+
- **Interface**: Only 3 database methods + 2 statement methods required
262+
- **No dependencies**: Users choose their own SQLite library
263+
- **100% compatible**: node:sqlite, better-sqlite3, @photostructure/sqlite all work
264+
- **Future-proof**: Any library matching the interface will work
265+
- **Type-safe**: TypeScript ensures compile-time compatibility

0 commit comments

Comments
 (0)