[core] Batch primary-key vector search over a shared plan#560
[core] Batch primary-key vector search over a shared plan#560JunRuiLee wants to merge 5 commits into
Conversation
c5d4766 to
2b4ca1d
Compare
2b4ca1d to
83da593
Compare
|
Add a batch entry to the bucket-local vector search so N query vectors share one pass over each ANN segment and each uncovered data file. The ANN searcher gains search_batch: it builds the live-row mask once (query-independent), opens one reader per segment, and drives visit_batch_vector_search, mapping results per query. The bucket search gains bucket_search_batch with per-query bounded heaps; a single query short-circuits to the existing single-query path so its result stays identical. The exact-fallback per-file search already scored all queries in one stream pass. Single-query wrappers are retained throughout.
Add a batch multi-query entry to the primary-key vector read: N query vectors share one snapshot/manifest plan, segment preload, per-file search closure and residual allow-list, then each query runs its own bucket search, exact rerank and merge before materializing into its own Arrow stream. The orchestrator gains search_candidates_batch; the builder factors the query-independent plan into a shared step reused across queries. BatchVectorSearchBuilder gains with_filter/with_projection and an execute_read returning one stream per query in input order (empty streams preserve arity, any query error fails the whole call); its scored execute stays fail-loud on the primary-key path, and a table without a primary-key vector index is rejected by the materialized read. The single-query entries become thin wrappers whose output is unchanged.
aa874b5 to
2607378
Compare
Thanks, fixed. Batch PK-vector query validation now also covers |
…dex dimension The branch's new up-front query-dimension guard validates every query against the vector column's dimension before planning and before the empty-plan early return. For an ARRAY<FLOAT> IVF_FLAT column with no explicit dimension, the resolver defaults to 128, so the dim-1 query vectors (`vec![1.0]`) in two guard tests failed loud before reaching the guards they exercise: - execute_read_empty_plan_reserved_projection_fails_loud - execute_read_filter_with_deletion_vectors_passes_guard Pin `fields.embedding.dimension = 4` and size the query vectors to match, so each test reaches its reserved-projection / deletion-vector residual guard as intended. Test-only; production behavior is unchanged.
87df59a to
a9c1478
Compare
|
1. Missing authorization check
2.
3. Default projection no longer rejects reserved-column conflicts The previous implementation validated the resolved field list after applying the projection, which also caught reserved-name conflicts in the default projection. The new resolver only validates explicitly requested projection names. As a result, a table containing a user column such as |
Three fixes from the PR review on the batch primary-key vector work: 1. The batch scored `execute()` now calls `ensure_read_authorized()` before any fast path, mirroring `execute_read` and the single-query builder. A `query-auth.enabled` table previously returned empty results on an empty snapshot and silently bypassed read authorization. 2. `pk_vector_query_dimension` resolves an `ARRAY<FLOAT>` column's dimension per the configured backend: Lumina via `LuminaVectorIndexOptions` (`lumina.index.dimension`), vindex/IVF via `VindexVectorIndexOptions`. A valid Lumina `ARRAY<FLOAT>` column no longer fails with "Unsupported vindex index type" before planning. 3. `resolve_materialize_read_type` validates the RESOLVED field list for reserved metadata column names, so the default (all user columns) projection also rejects a user column that collides with an injected metadata column, not only names in an explicit projection. Applied to both the single-query and batch resolvers. Adds a regression test for each.
The primary-key vector read injects its score column as `__paimon_search_score` (`SEARCH_SCORE_COLUMN`). `_PKEY_VECTOR_SCORE` was that column's earlier name; the rename left it behind in the reserved-name guard, two projection tests, and a few comments/assertion messages that named a column no longer produced anywhere. Remove `_PKEY_VECTOR_SCORE` from the reserved-column guard and the two projection tests, and correct the stale references to `__paimon_search_score` so the reserved set names only the three columns a read actually injects (`__paimon_search_score`, `_PKEY_VECTOR_POSITION`, `_ROW_ID`). No behavior change: nothing emits a `_PKEY_VECTOR_SCORE` column.
|
Addressed in fd72a9a (fixes) and 5fd4157 (cleanup): 1. Missing authorization check — 2. 3. Default projection reserved-column conflicts — both Separately, dropped the stale |
2 participants
Purpose
Part of #514, and the second of three PRs split out of the original combined change on the primary-key vector
execute_readpath. This PR adds batch multi-query search: N query vectors share one snapshot/manifest plan, segment preload, and opened readers. Single-query output stays byte-identical (single == batch-of-one).Brief change log
feat(vindex): the bucket-local ANN search gains a batch entry so N queries share one pass over each ANN segment and each uncovered file. The ANN searcher builds the live-row mask once (query-independent), opens one reader per segment, and drivesvisit_batch_vector_search; the exact searcher scores all queries in one stream pass. A single query short-circuits to the existing single-query path so its result is unchanged. The batch ANN scorer dispatches Lumina vs. vindex readers per segment backend, matching the single-query path.feat(table):BatchVectorSearchBuildergainswith_filter/with_projectionand anexecute_readreturning one Arrow stream per query in input order (empty streams preserve arity; any query error fails the whole call). The builder factors the query-independent plan (snapshot, segment preload, residual allow-list, backend, ANN scorer) into a shared step reused across queries. The single-queryplan_and_search_pk_candidatesbecomes a thin batch-of-one wrapper whose output is byte-identical. The scoredexecute()stays fail-loud on the primary-key path, and a data-evolutionexecute_readvalidates the target column exists and is a FLOAT vector column (ARRAY/VECTOR) up front so an unknown/scalar/non-float column fails loud instead of returning an empty stream.Tests
cargo test -p paimongreen;cargo build -p paimon-datafusionclean;cargo clippy -p paimon --all-targets -D warningsandcargo fmt --checkclean.API and Format
No on-disk format change, no new result columns. Single-query output is byte-identical. New public surface:
BatchVectorSearchBuilder::{with_filter, with_projection, execute_read}.Documentation
Code comments only; no user-facing docs change.