feat: Column statistics support v1#1916
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks for starting this. Collecting shuffle-time column statistics is a solid foundation for smarter broadcast and AQE decisions, and folding null counts across the input batches is a clean place to gather them. I left a few notes. The main one is that the collected stats are currently dropped at the scheduler, so it might be worth landing a consumer or a storage hook alongside the collection, or gating it behind a config until then. martin-g's note on python/Cargo.lock looks right, and it appears to be what is failing the Test Python Release jobs, since pyballista fails to compile against a mismatched DataFusion version. Reverting that lockfile to main and regenerating only if a dependency really changed should get those green.
This review was prepared with the help of an LLM.
|
Thank you . Let me address the PR comments and update here |
|
Created issue for part 2 : #1960 ( followup) |
Which issue does this PR close?
Closes #1827
Introduced new
TaskColumnStatsproto to compute null count , distinct values (TBD) for each column at the partition / stage level. Follow up PRs should leverage this information to make better decisions like broadcast / AQE etcRationale for this change
What changes are included in this PR?
Are there any user-facing changes?