feat: disable join dynamic filter pushdown by default#22438
Draft
adriangb wants to merge 1 commit into
Draft
Conversation
When a hash join's build-side dynamic filter contains a `hash_lookup` term, evaluating it on every probe-side row inside the scan duplicates the work the join's own probe is about to do. On TPC-H Q17 this doubles end-to-end query time (the equivalent of running the hash probe twice), and similar ~20-100% regressions show up across TPC-H Q3/Q5/Q8/Q9/Q14/Q18 and many TPC-DS queries that join a small dim table to a large fact table. Flip the default so users don't pay this cost on the common shape; the config is still available per-query when the build-side filter is selective enough to make scan-level pruning worthwhile (e.g. small dimension table that prunes most of a fact table's row groups / pages). The reset block at the end of dynamic_filter_pushdown_config.slt is switched from `SET ... = true` to `RESET ...` so the test ends in the configured default regardless of what that default is.
Contributor
Author
|
run benchmarks |
github-actions
Bot
added
documentation
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing disable-join-dynamic-filter-default (28c5c5a) to 50d74a7 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing disable-join-dynamic-filter-default (28c5c5a) to 50d74a7 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing disable-join-dynamic-filter-default (28c5c5a) to 50d74a7 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Flip
datafusion.optimizer.enable_join_dynamic_filter_pushdownfromtrue→falseby default.When a hash join's build-side dynamic filter contains a
hash_lookupterm, evaluating it on every probe-side row inside the scan duplicates the work the join's own probe is about to do. On TPC-H Q17 this doubles end-to-end query time, and similar ~20–100% regressions show up across TPC-H Q3/Q5/Q8/Q9/Q14/Q18 and many TPC-DS queries that join a small dim table to a large fact table.The config is still available per-query when the build-side filter is selective enough to make scan-level pruning worthwhile (e.g. small dim table that prunes most of a fact table's row groups / pages).
Benchmark numbers
From local SF1 / ClickBench-partitioned runs (12 vCPU), comparing
maindefaults vsmainwith this knob flipped tofalse:The total deltas are small in aggregate because the dynamic-filter pushdown both helps some queries (it does enable scan-level pruning) and hurts others (the doubled probe work). The default flip is about removing the regression tail; users who know their build side prunes well can re-enable per-query.
Test plan
cargo test --test sqllogictests— all 472 files pass after slt snapshot updates.run benchmarks