Skip to content

[BugFix] Reject outer WHERE on vectorSearch() subqueries to prevent silently dropped filters#1

Closed
mengweieric wants to merge 1 commit into
feature/vector-search-p0from
fix/vector-search-outer-where-subquery
Closed

[BugFix] Reject outer WHERE on vectorSearch() subqueries to prevent silently dropped filters#1
mengweieric wants to merge 1 commit into
feature/vector-search-p0from
fix/vector-search-outer-where-subquery

Conversation

@mengweieric

Copy link
Copy Markdown
Owner

Summary

Fixes the P0 tracker item Outer WHERE on subquery silently dropped from DSL: when vectorSearch() is wrapped in a subquery and the outer query has its own WHERE clause, the outer predicate never reaches the push-down contract. The plan shape after analysis is LogicalFilter over LogicalProject over the scan builder, so the PUSH_DOWN_FILTER rule (which matches filter directly above scanBuilder) does not fire. The filter is then applied in memory after k-NN has already returned top-k documents ranked by vector distance, which can silently yield zero rows with no explanation.

Failing shape

SELECT * FROM (SELECT v.firstname, v.state
               FROM vectorSearch(table='idx', field='emb',
                                 vector='[1,2]', option='k=5') AS v) t
WHERE t.state = 'TX'

Fix

Per the tracker (either reject vectorSearch() inside subqueries with outer WHERE, or document clearly) this PR chooses rejection as the safer preview behavior.

  • Adds a post-optimization validatePlan(LogicalPlan root) hook on TableScanBuilder (default no-op). Scan builders can now inspect their ancestors once all push-down rules have settled.
  • Wires it into Planner.plan() so every scan builder in the optimized plan gets a chance to validate.
  • VectorSearchIndexScanBuilder overrides the hook to walk the plan and reject the outer-WHERE-over-subquery shape: a LogicalFilter whose descendant chain reaches this scan builder through one or more LogicalProject nodes (the subquery-boundary marker). A filter directly above the scan builder (WHERE on vectorSearch() itself) is explicitly allowed because that path has already been exercised by push-down.
  • Error message names the shape and recommends moving the predicate inside the subquery so it is applied during the k-NN search.

Test plan

  • Unit tests in VectorSearchIndexScanBuilderTest covering the bad shape, nested subqueries, and three positive controls (filter directly on scanBuilder, inner filter wrapped in outer project, no filter at all) plus a bare-scanBuilder defensive baseline.
  • Integration tests in a new VectorSearchSubqueryIT covering the exact tracker query shape, the with-LIMIT variant, the _explain path, and positive controls for inner WHERE and no-WHERE subqueries.
  • Regression: VectorSearchIT (18/18) and VectorSearchExplainIT (11/11) still green.
  • ./gradlew spotlessCheck clean.
  • ./gradlew :core:test and :opensearch:test green.

When vectorSearch() is wrapped in a subquery and the outer query has its
own WHERE clause, for example:

  SELECT * FROM (SELECT v.firstname, v.state
                 FROM vectorSearch(table='idx', field='emb',
                                   vector='[1,2]', option='k=5') AS v) t
  WHERE t.state = 'TX'

the outer predicate never reaches the push-down contract. The plan shape
after analysis is LogicalFilter over LogicalProject over the scan
builder, so the PUSH_DOWN_FILTER rule (which matches filter directly
above scanBuilder) does not fire. The filter is then applied in memory
after k-NN has already returned top-k documents ranked by vector
distance, which can silently yield zero rows with no explanation.

This commit adds a post-optimization validation hook on TableScanBuilder
(default no-op) and invokes it from Planner.plan() once all push-down
rules have settled. VectorSearchIndexScanBuilder overrides the hook to
walk the fully optimized plan and reject the outer-WHERE-over-subquery
shape: a LogicalFilter whose descendant chain reaches the scan builder
through one or more LogicalProject nodes. A filter directly above the
scan builder (WHERE on vectorSearch() itself) is explicitly allowed
because that path has already been exercised by push-down.

The error message names the shape and tells the user to move the
predicate inside the subquery so it is applied during the k-NN search.

Tests:
 - VectorSearchIndexScanBuilderTest adds six cases covering the bad
   shape, nested subqueries, and the three positive controls (filter
   directly on scanBuilder, inner filter wrapped in outer project, no
   filter at all) plus a bare-scanBuilder defensive baseline.
 - VectorSearchSubqueryIT adds integration tests for the exact tracker
   query shape, the with-LIMIT variant, the _explain path, and positive
   controls for inner WHERE and no-WHERE subqueries.

Signed-off-by: Eric Wei <mengwei.eric@gmail.com>
@mengweieric mengweieric added bug Something isn't working enhancement New feature or request labels Apr 21, 2026
@mengweieric

Copy link
Copy Markdown
Owner Author

Closing and reopening against upstream opensearch-project/sql.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant