[core] Search primary-key vector buckets and files concurrently#556
Draft
JunRuiLee wants to merge 4 commits into
Draft
[core] Search primary-key vector buckets and files concurrently#556JunRuiLee wants to merge 4 commits into
JunRuiLee wants to merge 4 commits into
Conversation
JunRuiLee
force-pushed
the
feat/pk-vector-parallel-batch
branch
2 times, most recently
from
July 20, 2026 12:27
eb6bc77 to
f27a56d
Compare
JunRuiLee
changed the title
This was referenced Jul 20, 2026
Add a batch entry to the bucket-local vector search so N query vectors share one pass over each ANN segment and each uncovered data file. The ANN searcher gains search_batch: it builds the live-row mask once (query-independent), opens one reader per segment, and drives visit_batch_vector_search, mapping results per query. The bucket search gains bucket_search_batch with per-query bounded heaps; a single query short-circuits to the existing single-query path so its result stays identical. The exact-fallback per-file search already scored all queries in one stream pass. Single-query wrappers are retained throughout.
Add a batch multi-query entry to the primary-key vector read: N query vectors share one snapshot/manifest plan, segment preload, per-file search closure and residual allow-list, then each query runs its own bucket search, exact rerank and merge before materializing into its own Arrow stream. The orchestrator gains search_candidates_batch; the builder factors the query-independent plan into a shared step reused across queries. BatchVectorSearchBuilder gains with_filter/with_projection and an execute_read returning one stream per query in input order (empty streams preserve arity, any query error fails the whole call); its scored execute stays fail-loud on the primary-key path, and a table without a primary-key vector index is rejected by the materialized read. The single-query entries become thin wrappers whose output is unchanged.
…dex dimension The branch's new up-front query-dimension guard validates every query against the vector column's dimension before planning and before the empty-plan early return. For an ARRAY<FLOAT> IVF_FLAT column with no explicit dimension, the resolver defaults to 128, so the dim-1 query vectors (`vec![1.0]`) in two guard tests failed loud before reaching the guards they exercise: - execute_read_empty_plan_reserved_projection_fails_loud - execute_read_filter_with_deletion_vectors_passes_guard Pin `fields.embedding.dimension = 4` and size the query vectors to match, so each test reaches its reserved-projection / deletion-vector residual guard as intended. Test-only; production behavior is unchanged.
Add a global-index.thread-num option (default 32, rejected when non-positive) and thread its value as a concurrency limit through the primary-key vector search. The per-bucket loop in the orchestrator and the exact-fallback per-file loop in the bucket search now fan out with buffer_unordered up to that limit; a limit of 1 takes a plain sequential loop so file and bucket visit order stay deterministic. Results are accumulated by fixed query and split index and merged through the existing order-independent bounded heaps and global top-k, so the output does not depend on completion order. The ANN segment scan stays sequential because it is synchronous CPU work.
JunRuiLee
force-pushed
the
feat/pk-vector-parallel-batch
branch
from
July 21, 2026 14:10
f27a56d to
2836ae1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Part of #514, and the third of three PRs split out of the original combined change on the primary-key vector
execute_readpath. This PR adds intra-search parallelism: a configurable concurrency limit fans the per-bucket and per-exact-file searches out, without changing results.Brief change log
perf(table): aglobal-index.thread-numoption (default 32, rejected when non-positive) threads a concurrency limit through the search. The per-bucket loop (orchestrator) and per-exact-file loop (bucket search) fan out withbuffer_unorderedup to that limit; a limit of 1 takes a plain sequential loop so visit order stays deterministic. Results are accumulated by fixed query and split index and merged through the existing order-independent bounded heaps and global top-k, so output does not depend on completion order. The ANN segment scan stays sequential (synchronous CPU work).refactor(table): the exclusion predicate threaded through the search (residual filter ∩ deletion vectors) is markedSend + Syncso the search future isSendand can run on a multi-threaded runtime.Tests
global-index.thread-numdefault/explicit/non-positive.cargo test -p paimongreen;cargo build -p paimon-datafusionclean (cross-crateSendgate);cargo clippy -p paimon --all-targets -D warningsandcargo fmt --checkclean.API and Format
No on-disk format change, no new result columns. The concurrency limit defaults to a value that does not change results, only scheduling. New public surface: the
global-index.thread-numoption.Documentation
Code comments only; no user-facing docs change.