Skip to content

feat: Column statistics support v1#1916

Open
coderfender wants to merge 10 commits into
apache:mainfrom
coderfender:add_col_stats
Open

feat: Column statistics support v1#1916
coderfender wants to merge 10 commits into
apache:mainfrom
coderfender:add_col_stats

Conversation

@coderfender

@coderfender coderfender commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #1827

Introduced new TaskColumnStats proto to compute null count , distinct values (TBD) for each column at the partition / stage level. Follow up PRs should leverage this information to make better decisions like broadcast / AQE etc

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Comment thread python/Cargo.lock
@github-actions github-actions Bot removed the python label Jun 30, 2026
@coderfender coderfender marked this pull request as ready for review July 1, 2026 22:18

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for starting this. Collecting shuffle-time column statistics is a solid foundation for smarter broadcast and AQE decisions, and folding null counts across the input batches is a clean place to gather them. I left a few notes. The main one is that the collected stats are currently dropped at the scheduler, so it might be worth landing a consumer or a storage hook alongside the collection, or gating it behind a config until then. martin-g's note on python/Cargo.lock looks right, and it appears to be what is failing the Test Python Release jobs, since pyballista fails to compile against a mismatched DataFusion version. Reverting that lockfile to main and regenerating only if a dependency really changed should get those green.

This review was prepared with the help of an LLM.

Comment thread ballista/scheduler/src/state/execution_stage.rs
Comment thread ballista/core/src/execution_plans/sort_shuffle/writer.rs
Comment thread ballista/core/src/execution_plans/shuffle_writer.rs
@coderfender

Copy link
Copy Markdown
Contributor Author

Thank you . Let me address the PR comments and update here

@coderfender

Copy link
Copy Markdown
Contributor Author

Created issue for part 2 : #1960 ( followup)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve shuffle (column) statistics

3 participants