perf: prune irrelevant partitions task#1911
Open
coderfender wants to merge 11 commits into
Open
Conversation
andygrove
reviewed
Jun 28, 2026
Member
|
Thanks @coderfender. Could you fill out the PR description |
Co-authored-by: Andy Grove <agrove@apache.org>
martin-g
reviewed
Jun 29, 2026
andygrove
reviewed
Jul 2, 2026
andygrove
left a comment
andygrove
left a comment
Member
There was a problem hiding this comment.
Thanks for tackling this, the gRPC size ceiling at high partition counts is a real scaling limit and pruning per-task locations is a nice way to attack it. I left a few questions, mainly about the partition-alignment invariant and getting the Clippy build green. I skipped benchmarking since the change is off by default and the win is serialized-plan size rather than query time, which TPC-H at normal partition counts would not show.
This review was prepared with the help of an LLM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1880
Rationale for this change
What changes are included in this PR?
The current version in ballista sends out all partition information to every task on the executor. This becomes a bottleneck when the total partitions are high essentially reaching gRPC limit causing scale issues
The changes essentially eliminate redundant partition information being sent to the tasks. This is achieved by walking the plan and pruning partition id the task would be working on. There are special edge cases where a task works on more than ine partition (broadcast / coalesce) which are handled in the code
Added some unit tests as well
Are there any user-facing changes?
No
No