Skip to content

fix: impl handle_child_pushdown_result for SortExec#21527

Open
haohuaijin wants to merge 3 commits intoapache:mainfrom
haohuaijin:impl-handle-child
Open

fix: impl handle_child_pushdown_result for SortExec#21527
haohuaijin wants to merge 3 commits intoapache:mainfrom
haohuaijin:impl-handle-child

Conversation

@haohuaijin
Copy link
Copy Markdown
Contributor

@haohuaijin haohuaijin commented Apr 10, 2026

Which issue does this PR close?

Rationale for this change

FilterPushdown does not push filters through SortExec because SortExec lacks an implementation of handle_child_pushdown_result. When a FilterExec sits above a plain SortExec (no fetch), the filter can safely be moved below the sort without changing semantics, since sorting preserves all rows.

What changes are included in this PR?

Implemented handle_child_pushdown_result for SortExec in datafusion/physical-plan/src/sorts/sort.rs:

  • For plain SortExec (no fetch) in the Pre phase: any filters not absorbed by the child are collected into a new FilterExec inserted between the SortExec and its child.
  • For SortExec with fetch (TopK) or non-Pre phases: filters are not absorbed, preserving correct TopK semantics where filtering after limiting would change results.

Are these changes tested?

Added comprehensive tests in datafusion/core/tests/physical_optimizer/filter_pushdown.rs covering:

  • Filter pushdown through sort into a scan that supports pushdown
  • Filter pushdown through sort when scan does NOT support pushdown (FilterExec inserted between sort and scan)
  • Multiple conjunctive filters pushed through sort
  • Filter NOT pushed through sort with fetch (TopK)
  • Filter with projection pushed through sort
  • Filter pushdown preserving the preserve_partitioning flag
  • Filter with fetch limit propagated to SortExec (TopK conversion)

and one test case test_filter_pushdown_through_sort_with_projection for use LogicalPlanBuilder to reproduce.
not able to use sql to reproduce

Are there any user-facing changes?

@github-actions github-actions bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Apr 10, 2026
@haohuaijin
Copy link
Copy Markdown
Contributor Author

the reproduce result in #21526, for this pr

Physical plan:
FilterExec: log@1 LIKE %datafusion%, projection=[_timestamp@0]
  SortExec: expr=[_timestamp@0 DESC NULLS LAST], preserve_partitioning=[true]
    DataSourceExec: partitions=1, partition_sizes=[1]

After FilterPushdown:
ProjectionExec: expr=[_timestamp@0 as _timestamp]
  SortExec: expr=[_timestamp@0 DESC NULLS LAST], preserve_partitioning=[true]
    FilterExec: log@1 LIKE %datafusion%
      DataSourceExec: partitions=1, partition_sizes=[1]

@haohuaijin haohuaijin changed the title fix: impl handle_child_pushdown_result for SortExec fix: impl handle_child_pushdown_result for SortExec Apr 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

filter not pushdown through sort in FilterPushdown

1 participant