Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The sort-based shuffle writer is ~50x faster than the current hash-based shuffler writer for real-world workloads. However, we do not yet support IO_BLOCK_TRANSPORT for sort-based shuffle, and the flight path is very inefficient since it involves re-encoding batches.
Describe the solution you'd like
Modify the sort-based shuffle disk format so that we can enable IO_BLOCK_TRANSPORT. This is how it is implemented in Comet already.
Describe alternatives you've considered
Additional context
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The sort-based shuffle writer is ~50x faster than the current hash-based shuffler writer for real-world workloads. However, we do not yet support IO_BLOCK_TRANSPORT for sort-based shuffle, and the flight path is very inefficient since it involves re-encoding batches.
Describe the solution you'd like
Modify the sort-based shuffle disk format so that we can enable IO_BLOCK_TRANSPORT. This is how it is implemented in Comet already.
Describe alternatives you've considered
Additional context