perf: address scheduler crash on high partition SF data#1875

coderfender wants to merge 5 commits into

apache:mainapache/datafusion-ballista:mainfrom

coderfender:make_partition_metadata_leanercoderfender/datafusion-ballista:make_partition_metadata_leanerCopy head branch name to clipboard