Skip to content

Commit 28c72d8

Browse files
SubhamSinghalSubhamSinghal
committed
Merge push down topk with push down limit rule
1 parent 216be5e commit 28c72d8

5 files changed

Lines changed: 386 additions & 522 deletions

File tree

‎datafusion/core/src/optimizer_rule_reference.md‎

Lines changed: 28 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -35,34 +35,34 @@ Rule order matters. The default pipeline may change between releases.
3535

3636
### Logical Optimizer Rules
3737

38-
| order | rule | summary |
39-
| ----- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
40-
| 1 | `rewrite_set_comparison` | Rewrites `ANY` and `ALL` set-comparison subqueries into `EXISTS`-based boolean expressions with correct SQL NULL semantics. |
41-
| 2 | `optimize_unions` | Flattens nested unions and removes unions with a single input. |
42-
| 3 | `unions_to_filter` | Merges `UNION DISTINCT` branches that share the same source into a single filtered branch with a disjunctive predicate. |
43-
| 4 | `simplify_expressions` | Constant-folds and simplifies expressions while preserving output names. |
44-
| 5 | `replace_distinct_aggregate` | Rewrites `DISTINCT` and `DISTINCT ON` operators into aggregate-based plans that later rules can optimize further. |
45-
| 6 | `eliminate_join` | Replaces keyless inner joins with a literal `false` filter by an empty relation. |
46-
| 7 | `decorrelate_predicate_subquery` | Converts eligible `IN` and `EXISTS` predicate subqueries into semi or anti joins. |
47-
| 8 | `scalar_subquery_to_join` | Rewrites eligible scalar subqueries into joins and adds schema-preserving projections. |
48-
| 9 | `decorrelate_lateral_join` | Rewrites eligible lateral joins into regular joins. |
49-
| 10 | `extract_equijoin_predicate` | Splits join filters into equijoin keys and residual predicates. |
50-
| 11 | `eliminate_duplicated_expr` | Removes duplicate expressions from projections, aggregates, and similar operators. |
51-
| 12 | `eliminate_filter` | Drops always-true filters and replaces always-false or NULL filters with empty relations. |
52-
| 13 | `eliminate_cross_join` | Uses filter predicates to replace cross joins with inner joins when join keys can be found. |
53-
| 14 | `eliminate_limit` | Removes no-op limits and simplifies trivial limit shapes. |
54-
| 15 | `propagate_empty_relation` | Pushes empty-relation knowledge upward so operators fed by no rows collapse early. |
55-
| 16 | `filter_null_join_keys` | Adds `IS NOT NULL` filters to nullable equijoin keys that can never match. |
56-
| 17 | `eliminate_outer_join` | Rewrites outer joins to inner joins when later filters reject the NULL-extended rows. |
57-
| 18 | `push_down_limit` | Moves literal limits closer to scans and unions and merges adjacent limits. |
58-
| 19 | `push_down_filter` | Moves filters as early as possible through filter-commutative operators. |
59-
| 20 | `push_down_topk_through_join` | Pushes Sort with LIMIT through joins when sort columns come from the preserved side. |
60-
| 21 | `single_distinct_aggregation_to_group_by` | Rewrites single-column `DISTINCT` aggregations into two-stage `GROUP BY` plans. |
61-
| 22 | `eliminate_group_by_constant` | Removes constant or functionally redundant expressions from `GROUP BY`. |
62-
| 23 | `common_sub_expression_eliminate` | Computes repeated subexpressions once and reuses the result. |
63-
| 24 | `extract_leaf_expressions` | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. |
64-
| 25 | `push_down_leaf_projections` | Pushes the helper projections created by leaf extraction toward leaf inputs. |
65-
| 26 | `optimize_projections` | Prunes unused columns and removes unnecessary logical projections. |
38+
| order | rule | summary |
39+
| ----------------------------------------------------------------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
40+
| 1 | `rewrite_set_comparison` | Rewrites `ANY` and `ALL` set-comparison subqueries into `EXISTS`-based boolean expressions with correct SQL NULL semantics. |
41+
| 2 | `optimize_unions` | Flattens nested unions and removes unions with a single input. |
42+
| 3 | `unions_to_filter` | Merges `UNION DISTINCT` branches that share the same source into a single filtered branch with a disjunctive predicate. |
43+
| 4 | `simplify_expressions` | Constant-folds and simplifies expressions while preserving output names. |
44+
| 5 | `replace_distinct_aggregate` | Rewrites `DISTINCT` and `DISTINCT ON` operators into aggregate-based plans that later rules can optimize further. |
45+
| 6 | `eliminate_join` | Replaces keyless inner joins with a literal `false` filter by an empty relation. |
46+
| 7 | `decorrelate_predicate_subquery` | Converts eligible `IN` and `EXISTS` predicate subqueries into semi or anti joins. |
47+
| 8 | `scalar_subquery_to_join` | Rewrites eligible scalar subqueries into joins and adds schema-preserving projections. |
48+
| 9 | `decorrelate_lateral_join` | Rewrites eligible lateral joins into regular joins. |
49+
| 10 | `extract_equijoin_predicate` | Splits join filters into equijoin keys and residual predicates. |
50+
| 11 | `eliminate_duplicated_expr` | Removes duplicate expressions from projections, aggregates, and similar operators. |
51+
| 12 | `eliminate_filter` | Drops always-true filters and replaces always-false or NULL filters with empty relations. |
52+
| 13 | `eliminate_cross_join` | Uses filter predicates to replace cross joins with inner joins when join keys can be found. |
53+
| 14 | `eliminate_limit` | Removes no-op limits and simplifies trivial limit shapes. |
54+
| 15 | `propagate_empty_relation` | Pushes empty-relation knowledge upward so operators fed by no rows collapse early. |
55+
| 16 | `filter_null_join_keys` | Adds `IS NOT NULL` filters to nullable equijoin keys that can never match. |
56+
| 17 | `eliminate_outer_join` | Rewrites outer joins to inner joins when later filters reject the NULL-extended rows. |
57+
| 18 | `push_down_limit` | Moves literal limits closer to scans and unions and merges adjacent limits, and pushes |
58+
| `Sort(fetch=N)` onto a join's preserved-side child for LEFT/RIGHT/CROSS/MARK joins. |
59+
| 19 | `push_down_filter` | Moves filters as early as possible through filter-commutative operators. |
60+
| 20 | `single_distinct_aggregation_to_group_by` | Rewrites single-column `DISTINCT` aggregations into two-stage `GROUP BY` plans. |
61+
| 21 | `eliminate_group_by_constant` | Removes constant or functionally redundant expressions from `GROUP BY`. |
62+
| 22 | `common_sub_expression_eliminate` | Computes repeated subexpressions once and reuses the result. |
63+
| 23 | `extract_leaf_expressions` | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. |
64+
| 24 | `push_down_leaf_projections` | Pushes the helper projections created by leaf extraction toward leaf inputs. |
65+
| 25 | `optimize_projections` | Prunes unused columns and removes unnecessary logical projections. |
6666

6767
### Physical Optimizer Rules
6868

‎datafusion/optimizer/src/lib.rs‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,6 @@ pub mod optimizer;
6565
pub mod propagate_empty_relation;
6666
pub mod push_down_filter;
6767
pub mod push_down_limit;
68-
pub mod push_down_topk_through_join;
6968
pub mod replace_distinct_aggregate;
7069
pub mod rewrite_set_comparison;
7170
pub mod scalar_subquery_to_join;

‎datafusion/optimizer/src/optimizer.rs‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,6 @@ use crate::plan_signature::LogicalPlanSignature;
6060
use crate::propagate_empty_relation::PropagateEmptyRelation;
6161
use crate::push_down_filter::PushDownFilter;
6262
use crate::push_down_limit::PushDownLimit;
63-
use crate::push_down_topk_through_join::PushDownTopKThroughJoin;
6463
use crate::replace_distinct_aggregate::ReplaceDistinctWithAggregate;
6564
use crate::rewrite_set_comparison::RewriteSetComparison;
6665
use crate::scalar_subquery_to_join::ScalarSubqueryToJoin;
@@ -309,7 +308,6 @@ impl Optimizer {
309308
// Filters can't be pushed down past Limits, we should do PushDownFilter after PushDownLimit
310309
Arc::new(PushDownLimit::new()),
311310
Arc::new(PushDownFilter::new()),
312-
Arc::new(PushDownTopKThroughJoin::new()),
313311
Arc::new(SingleDistinctToGroupBy::new()),
314312
// The previous optimizations added expressions and projections,
315313
// that might benefit from the following rules

0 commit comments

Comments
 (0)