Commit 7456b23
committed
perf(table): stream primary-key vector exact fallback one batch at a time
Replace the whole-column preload of an uncovered data file's vector column
with a per-file search that streams the column one Arrow batch at a time into
per-query bounded heaps, so peak memory is one batch plus the top-k heap rather
than the full column. The per-file exact-fallback factory becomes a search
closure (Fn + Send + Sync) returning one bounded best-first list per query;
the streaming loop lives in the table layer while the bucket search only calls
the closure and merges its results. Queries are validated before any file
stream is opened, and the drained row count is checked against the file
metadata in both directions.1 parent 4211dda commit 7456b23
6 files changed
Lines changed: 929 additions & 484 deletions
File tree
- crates/paimon/src
- table
- vindex/pkvector
0 commit comments