ST_Intersects pushdown: vortex.geo.intersects kernel, DuckDB lowering, spatial function overrides#8704
Open
HarukiMoriarty wants to merge 4 commits into
Open
ST_Intersects pushdown: vortex.geo.intersects kernel, DuckDB lowering, spatial function overrides#8704HarukiMoriarty wants to merge 4 commits into
HarukiMoriarty wants to merge 4 commits into
Conversation
OGC intersects (not disjoint; boundary contact counts) between two native geometry operands, each a column or a constant literal. Constant pairs fold to a ConstantArray; column shapes materialize geo_types rows and delegate to geo::Intersects. The function id is kept stable so a future stats-pruning rewrite can key on it. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
geo computes row by row, so every kernel materializes geo_types geometries from the columnar storage first; the hand-rolled point fast paths were fragments of a columnar geometry compute library that does not exist yet. Keep the kernels as thin materialize-and-delegate wrappers until that library exists (noted in scalar_fn/mod.rs); on SpatialBench SF1 the pushed Q1 filter goes from 5.4ms to 12.6ms, still well ahead of the unpushed path. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
A two-argument st_intersects over a native geometry column and a constant GEOMETRY literal (WKB, decoded to its native scalar at plan time) now pushes into the Vortex scan via pushdown_complex_filter, like st_distance and st_dwithin. The shared operand extraction moves into a two_geo_operands helper and the lowered-name list lives only in the match. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
DuckDB refuses to push can-throw filters through a projection (pushdown_projection.cpp), spatial marks ST_Intersects fallible, and every view-wrapped scan sits under a projection, so those filters never reached the Vortex scan. Shadow the function with a copy whose error mode is cleared, the same trick the ST_DWithin override uses for its folded radius. The overrides are now table-driven: spatial_overrides.hpp holds one (name, arity, tweak) row per shadowed function, shared by the single registration entry point (duckdb_vx_register_spatial_overrides) and the join-condition restore pass (RestoreSpatialOverrides, formerly RestoreStDWithin). Adding an override is one table row plus a lowering arm. On SpatialBench SF1 the view-wrapped 6M-row intersects probe drops from 406ms (DuckDB scalar filter) to the pushed in-scan path, and Q6's zone-side conjunct now pushes while its ST_Within spatial join stays intact; Q1-Q9 row counts validate. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | rebuild_naive |
91.4 µs | 109 µs | -16.17% |
| ❌ | Simulation | chunked_varbinview_opt_canonical_into[(100, 100)] |
305.4 µs | 340.6 µs | -10.32% |
| ⚡ | Simulation | chunked_varbinview_canonical_into[(1000, 10)] |
191 µs | 154.7 µs | +23.42% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[128] |
273.6 ns | 244.4 ns | +11.93% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing nemo/geo-intersects (32dd2cc) with develop (1bcaf3f)
Footnotes
-
42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
myrrc
requested changes
Jul 10, 2026
| class StDWithinRestore final : public LogicalOperatorVisitor { | ||
| // Whether `expr` is a call bound through one of the Vortex user-catalog overrides (see | ||
| // `SpatialOverrides` in spatial_overrides.hpp). | ||
| bool IsSpatialOverrideCall(const BoundFunctionExpression &expr) { |
Contributor
There was a problem hiding this comment.
Let's move this code to a separate translation unit spatial_overrides.hpp/cpp as it's not really related to a scalar function pushdown, but overriding some of functions.
Moreover, when 2.0 comes it would be harder for me to remove scalar functions pushdown (upstreamed).
|
|
||
| /// Shadow spatial's `fn_override.name` in the user catalog with tweaked copies of its overloads; | ||
| /// a no-op when the function is not registered (`spatial` not loaded). | ||
| static void RegisterSpatialOverride(ClientContext &context, const SpatialOverride &fn_override) { |
Contributor
There was a problem hiding this comment.
Let's also move this code to a spatial translation unit
Contributor
|
Let's add a sqllogictest which checks our queries don't break if we override the functions |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
ST_Intersectsover native geometry columns currently evaluates inside DuckDB, exporting every row asGEOMETRYjust to be tested. This adds a nativevortex.geo.intersectskernel and pushes the filter into the scan: on SpatialBench SF1 (6M points vs a literal polygon), 406ms (DuckDB filter) / 36.6ms (SPATIAL_JOIN) -> 12.4ms pushed. Part of the native geospatial lane, following theST_Distance/ST_DWithinpushdown.What changes are included in this PR?
vortex-geo:GeoIntersectskernel.vortex-duckdb: lowerst_intersects(native column, GEOMETRY literal)likest_distance.vortex-duckdb: shadow spatial'sST_Intersectswith a non-fallible copy — DuckDB won't push can-throw filters through the projection every view-registered table has. TheST_DWithinoverride generalizes into a table-driven registry (spatial_overrides.hpp) shared by registration and the join-condition restore pass.Note: drop
GeoDistance's columnar point fast paths.geois row-oriented, so kernels materialize rows regardless; hand-rolled columnar paths belong in a future columnar geometry compute library (pushed Q1: 5.4ms -> 12.6ms, still far ahead of unpushed).