perf: coalesce batches before sending to distributor channels in RepartitionExec#22010
perf: coalesce batches before sending to distributor channels in RepartitionExec#22010
Conversation
…butor channel, instead of after.
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing gabotechs/repartition-coalesce-batches-berfore-channels (dcf6482) to b2fd2d3 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
🙈 Don't look yet, still experimenting
Which issue does this PR close?
Rationale for this change
Today RepartitionExec emits one small batch per (input batch × non-empty output partition), then coalesces those small batches back into target-sized ones on the consumer side. That means the channel layer (memory accounting, gate, await suspensions) does work proportional to num_partitions per input batch, even though each small batch only carries batch_size / num_partitions rows. Moving the coalescing producer-side, before the gate, collapses the channel traffic and reduces sensitivity to high output fanout.
What changes are included in this PR?
Are these changes tested?
Yes, by existing tests
Are there any user-facing changes?
No