Remove chunked special case from stat execution#7928
Conversation
Merging this PR will degrade performance by 17.6%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | new_bp_prim_test_between[i32, 16384] |
95.5 µs | 109.2 µs | -12.52% |
| ❌ | Simulation | new_bp_prim_test_between[i32, 32768] |
141.9 µs | 170.1 µs | -16.58% |
| ❌ | Simulation | new_bp_prim_test_between[i64, 32768] |
178.9 µs | 236.8 µs | -24.48% |
| ❌ | Simulation | new_bp_prim_test_between[i64, 16384] |
116 µs | 144.6 µs | -19.77% |
| ❌ | Simulation | new_alp_prim_test_between[f64, 16384] |
127.8 µs | 148.8 µs | -14.09% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ngates/stats-7707/chunked-stat-execute-parent (0dce6a1) with develop (95f429d)
Footnotes
-
24 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
07bfcdc to
261cbe4
Compare
| let input = args.get(0)?; | ||
| let dtype = stat_dtype(options.aggregate_fn(), input.dtype())?; | ||
|
|
||
| // Recurse into each chunk so the output keeps per-chunk granularity (one constant |
There was a problem hiding this comment.
I realized we already have a rule that pushes scalar functions through chunked arrays
Signed-off-by: Nicholas Gates <nick@nickgates.com>
261cbe4 to
0dce6a1
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals ProfilingVortex (geomean): 1.145x ❌ datafusion / vortex-file-compressed (1.145x ❌, 0↑ 8↓)
|
File Sizes: PolarSignals ProfilingNo file size changes detected. |
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.928x ➖, 3↑ 0↓)
datafusion / vortex-compact (0.945x ➖, 1↑ 0↓)
datafusion / parquet (0.919x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed (0.907x ➖, 2↑ 0↓)
duckdb / vortex-compact (0.963x ➖, 1↑ 0↓)
duckdb / parquet (0.935x ➖, 1↑ 0↓)
Full attributed analysis
|
File Sizes: FineWeb NVMeNo file size changes detected. |
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (1.029x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.024x ➖, 0↑ 0↓)
datafusion / parquet (0.965x ➖, 2↑ 1↓)
datafusion / arrow (1.003x ➖, 2↑ 1↓)
duckdb / vortex-file-compressed (1.030x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.040x ➖, 0↑ 0↓)
duckdb / parquet (1.007x ➖, 0↑ 2↓)
duckdb / duckdb (1.027x ➖, 1↑ 0↓)
Full attributed analysis
|
File Sizes: TPC-H SF=1 on NVMENo file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (1.005x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.007x ➖, 1↑ 3↓)
datafusion / parquet (0.999x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed (0.999x ➖, 0↑ 3↓)
duckdb / vortex-compact (0.993x ➖, 1↑ 0↓)
duckdb / parquet (0.996x ➖, 0↑ 0↓)
duckdb / duckdb (1.000x ➖, 1↑ 1↓)
Full attributed analysis
|
File Sizes: TPC-DS SF=1 on NVMENo file size changes detected. |
Benchmarks: FineWeb S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (1.033x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.070x ➖, 0↑ 1↓)
datafusion / parquet (1.064x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.006x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.038x ➖, 0↑ 1↓)
duckdb / parquet (1.021x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: Random AccessVortex (geomean): 0.994x ➖ unknown / unknown (1.010x ➖, 1↑ 1↓)
|
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) duckdb / vortex-file-compressed (1.006x ➖, 0↑ 1↓)
duckdb / vortex-compact (1.032x ➖, 0↑ 1↓)
duckdb / parquet (0.991x ➖, 0↑ 0↓)
Full attributed analysis
|
File Sizes: Statistical and Population GeneticsNo file size changes detected. |
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.832x ✅, 21↑ 0↓)
datafusion / vortex-compact (0.843x ✅, 22↑ 0↓)
datafusion / parquet (0.892x ✅, 13↑ 0↓)
datafusion / arrow (0.849x ✅, 20↑ 0↓)
duckdb / vortex-file-compressed (0.914x ➖, 7↑ 0↓)
duckdb / vortex-compact (0.871x ✅, 20↑ 0↓)
duckdb / parquet (0.935x ➖, 2↑ 0↓)
duckdb / duckdb (0.922x ➖, 5↑ 0↓)
Full attributed analysis
|
File Sizes: TPC-H SF=10 on NVMENo file size changes detected. |
Benchmarks: Clickbench on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (1.011x ➖, 0↑ 2↓)
datafusion / parquet (1.009x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.979x ➖, 6↑ 0↓)
duckdb / parquet (1.002x ➖, 0↑ 1↓)
duckdb / duckdb (1.022x ➖, 0↑ 3↓)
Full attributed analysis
|
File Sizes: Clickbench on NVMEFile Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (0.983x ➖, 2↑ 2↓)
datafusion / vortex-compact (1.120x ➖, 1↑ 5↓)
datafusion / parquet (0.990x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.993x ➖, 0↑ 1↓)
duckdb / vortex-compact (0.954x ➖, 0↑ 0↓)
duckdb / parquet (1.006x ➖, 0↑ 1↓)
Full attributed analysis
|
Benchmarks: CompressionVortex (geomean): 0.988x ➖ unknown / unknown (0.971x ➖, 12↑ 2↓)
|
Benchmarks: TPC-H SF=10 on S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (1.068x ➖, 0↑ 3↓)
datafusion / vortex-compact (1.004x ➖, 1↑ 2↓)
datafusion / parquet (1.123x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed (0.962x ➖, 0↑ 1↓)
duckdb / vortex-compact (0.940x ➖, 0↑ 0↓)
duckdb / parquet (0.899x ➖, 0↑ 0↓)
Full attributed analysis
|
Part of #7707.
Stack: 1/3
Base:
developNext: #7929
Summary
StatFn::execute.Checks
cargo test -p vortex-array stat_expr_reads_cached_sum_per_chunk