Benchmark single-column GROUP BY: hash vs direct-indexed flat array#22481
Benchmark single-column GROUP BY: hash vs direct-indexed flat array#22481nathanb9 wants to merge 1 commit into
Conversation
Adds `GroupValuesFlatPrimitive`, a direct-indexing GroupValues implementation for integer-typed GROUP BY columns with bounded value ranges. Uses `value - min` to index directly into a flat array — no hashing, no collisions, O(1) lookup. Adds a benchmark (`single_group_by_primitive`) that directly compares hash-based `GroupValuesPrimitive` vs `GroupValuesFlatPrimitive` using the same `iter_batched_ref` methodology as the multi-column GROUP BY benchmark in apache#22322. Only `intern()` is timed; construction is in setup. Three experiments: 1. Group count sweep (10–100K groups, 1M rows) 2. Density sweep (10K groups, 10%–100% density) 3. Row count scaling (10K groups, 1M–10M rows) Closes benchmarking aspect of apache#19938. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing benchmark-single-group-by-flat-primitive (c2c07a4) to b382ecd (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing benchmark-single-group-by-flat-primitive (c2c07a4) to b382ecd (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing benchmark-single-group-by-flat-primitive (c2c07a4) to b382ecd (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Summary
GroupValuesFlatPrimitive, a direct-indexingGroupValuesimplementation for integer GROUP BY columns with bounded value ranges (no hashing, O(1) array lookup viavalue - min)single_group_by_primitive) comparing hash-basedGroupValuesPrimitivevsGroupValuesFlatPrimitivesingle_group_byandprimitivemodules public for benchmark accessBenchmark design
Uses the same
iter_batched_refmethodology as the multi-column GROUP BY benchmark in #22322:intern()calls are measuredblack_boxprevents dead-code eliminationThree experiments:
Local results (Apple Silicon, release mode,
iter_batched_ref)Related
ArrayMapinjoins/array_map.rs(perfect hash join, PR perfect hash join #19411)Test plan
cargo test -p datafusion-physical-plan --lib flat_primitive— 5 unit tests passcargo clippy -p datafusion-physical-plan --benches -- -D warnings— cleancargo bench -p datafusion-physical-plan --bench single_group_by_primitive— runs successfully🤖 Generated with Claude Code + reviewed by me 😃