fix: Batch size limit in re-spill compounds (apache#23286)

This fixes an issue in my previous design for the re-spill in sort -
whenever we half a stream, we half self.batch_size, even if its not the
same stream being halved, so for each re-spill we get a smaller and
smaller output batch, which really hurts performance in the merge.
We can avoid it by just keeping the batch_size_limit in the spilled file
struct. then we just use the lowest limit whenever we merge N streams,
making it dynamic and much more robust.

Has tests.

Does not break API

---------

Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com>

Author: EmilyMatt