Commit f460fae
committed
feat: add ParquetLookupProvider and SqliteLookupProvider behind feature gates
Ports the two storage backends proven in the df-vector-search benchmark POC
into the library as optional, feature-gated providers.
- `parquet-provider` feature: ParquetLookupProvider — concurrent row-group
reads from any ObjectStore (S3, local FS) with pre-cached parquet footers
and optional RowSelection for page-skip optimisation
- `sqlite-provider` feature: SqliteLookupProvider — B-tree point lookups via
a WAL-mode connection pool; builds from parquet on first run
- `keys` module (always compiled): pack_key / unpack_key / DatasetLayout —
shared key encoding utilities extracted from indexing.rs
- Integration tests for both providers (9 tests total)
Breaking change in Cargo.toml: tokio gains the "sync" feature unconditionally
(needed by SqliteLookupProvider's Semaphore; tokio was already a hard dep).1 parent 64a7b1c commit f460fae
8 files changed
Lines changed: 1522 additions & 49 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
8 | 12 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
13 | 17 | | |
14 | | - | |
15 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
16 | 31 | | |
17 | 32 | | |
18 | | - | |
| 33 | + | |
| 34 | + | |
0 commit comments