Skip to content

Commit 7456b23

Browse files
committed
perf(table): stream primary-key vector exact fallback one batch at a time
Replace the whole-column preload of an uncovered data file's vector column with a per-file search that streams the column one Arrow batch at a time into per-query bounded heaps, so peak memory is one batch plus the top-k heap rather than the full column. The per-file exact-fallback factory becomes a search closure (Fn + Send + Sync) returning one bounded best-first list per query; the streaming loop lives in the table layer while the bucket search only calls the closure and merges its results. Queries are validated before any file stream is opened, and the drained row count is checked against the file metadata in both directions.
1 parent 4211dda commit 7456b23

6 files changed

Lines changed: 929 additions & 484 deletions

File tree

0 commit comments

Comments
 (0)