Skip to content

Add ListMin and ListMax expressions#8856

Open
mhk197 wants to merge 3 commits into
developfrom
mk/list-min-max
Open

Add ListMin and ListMax expressions#8856
mhk197 wants to merge 3 commits into
developfrom
mk/list-min-max

Conversation

@mhk197

@mhk197 mhk197 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Add ListMin and ListMax scalar functions and expression constructors, with explicit null and NaN semantics and a performant primitive grouped kernel.

What changes are included in this PR?

  • Add ListMin and ListMax scalar functions and expression constructors.
  • Support List, ListView, and FixedSizeList inputs
  • Support bool, primitive, decimal, UTF-8, and binary elements; reject nested and extension ordering until their semantics are explicit.
  • Add a primitive grouped extrema kernel.

Final performance

Apple Silicon release medians over 8,192 nullable i32 lists:

Function List size DataFusion/Arrow Vortex Vortex advantage
minimum 10 57.16 us 53.74 us 1.06x
minimum 100 372.5 us 263.9 us 1.41x
minimum 1,000 1.936 ms 0.900 ms 2.15x
maximum 10 63.91 us 53.16 us 1.20x
maximum 100 390.7 us 262.1 us 1.49x
maximum 1,000 1.993 ms 0.902 ms 2.21x

@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 1795 untouched benchmarks
🆕 18 new benchmarks
⏩ 44 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation compact[(4096, 90)] 780.3 ns 867.8 ns -10.08%
Simulation fsl_large 99.6 µs 89 µs +11.98%
Simulation compress[(100000, 4)] 1.8 ms 1.6 ms +11.94%
🆕 Simulation vortex_fixed_size_list_max[(8192, 10)] N/A 788.7 µs N/A
🆕 Simulation vortex_fixed_size_list_max[(8192, 100)] N/A 3.3 ms N/A
🆕 Simulation vortex_fixed_size_list_max[(8192, 1000)] N/A 20.6 ms N/A
🆕 Simulation vortex_fixed_size_list_min[(8192, 10)] N/A 833.2 µs N/A
🆕 Simulation vortex_fixed_size_list_min[(8192, 100)] N/A 3.3 ms N/A
🆕 Simulation vortex_fixed_size_list_min[(8192, 1000)] N/A 20.6 ms N/A
🆕 Simulation vortex_list_max[(8192, 10)] N/A 1 ms N/A
🆕 Simulation vortex_list_max[(8192, 100)] N/A 3.5 ms N/A
🆕 Simulation vortex_list_max[(8192, 1000)] N/A 20.9 ms N/A
🆕 Simulation vortex_list_min[(8192, 10)] N/A 1.1 ms N/A
🆕 Simulation vortex_list_min[(8192, 100)] N/A 3.5 ms N/A
🆕 Simulation vortex_list_min[(8192, 1000)] N/A 20.9 ms N/A
🆕 Simulation vortex_listview_max[(8192, 10)] N/A 1 ms N/A
🆕 Simulation vortex_listview_max[(8192, 100)] N/A 3.4 ms N/A
🆕 Simulation vortex_listview_max[(8192, 1000)] N/A 21.3 ms N/A
🆕 Simulation vortex_listview_min[(8192, 10)] N/A 1 ms N/A
🆕 Simulation vortex_listview_min[(8192, 100)] N/A 3.4 ms N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/list-min-max (716d31e) with develop (add6944)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 716d31e 1 Explore Profiling Data
Previous Runs (3)
Status Commit Job Attempt Link
🟢 Done 8ec5fd1 1 Explore Profiling Data
🟢 Done 6087918 1 Explore Profiling Data
🟢 Done 7e6519d 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +5.0%
Engines: DataFusion Likely regression (+10.4%, medium confidence) · DuckDB No clear signal (-0.0%, low confidence)
Vortex (geomean): 0.969x ➖
Parquet (geomean): 0.946x ➖
Shifts: Parquet (control) -5.4% · Median polish -1.1%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (0.999x ➖, 0↑ 0↓)
name PR 716d31e (ns) base 5b688a9 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 10552170 10644295 0.99
vortex_q01/datafusion:vortex-file-compressed 6871750 6830634 1.01
datafusion / parquet (0.905x ➖, 1↑ 0↓)
name PR 716d31e (ns) base 5b688a9 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20858585 22843824 0.91
vortex_q01/datafusion:parquet 🚀 4875941 5437392 0.90
duckdb / vortex-file-compressed (0.989x ➖, 0↑ 0↓)
name PR 716d31e (ns) base 5b688a9 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10975406 10947785 1.00
vortex_q01/duckdb:vortex-file-compressed 6777044 6952563 0.97
duckdb / parquet (0.989x ➖, 0↑ 0↓)
name PR 716d31e (ns) base 5b688a9 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23958025 24229204 0.99
vortex_q01/duckdb:parquet 9637740 9746132 0.99

No file size changes detected.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 force-pushed the mk/list-min-max branch from 7e6519d to 6087918 Compare July 20, 2026 14:34
@mhk197 mhk197 changed the title feat: add list min and max expressions Add ListMin and ListMax expressions Jul 20, 2026
@mhk197 mhk197 added the changelog/feature A new feature label Jul 20, 2026
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 marked this pull request as ready for review July 20, 2026 15:02
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
connortsui20 added a commit that referenced this pull request Jul 20, 2026
## Rationale for this change

- Closes: #8807

Three benchmarks stayed flaky after #8742, flipping between the same two
values on PRs that can't affect them. Same root causes and fixes as
#8742:

| Benchmark | Seen flaky on | Why | Fix |
| --- | --- | --- | --- |
| `true_count_vortex_buffer[128]` | ±11.17% on 9 unrelated PRs (#8805,
#8811, #8812, #8820, #8843, #8803, …) | a 128-bit popcount measures
harness overhead and code layout, not the count | drop the 128 size |
| runend `compress[(100000, 4)]` | ±11.9% on #8805, #8750, #8856 |
allocates in the timed region; glibc malloc differs across runner images
| mimalloc as global allocator |
| `cast_decimal` `copy_*[65536]` | identical flags on #8838 and #8724 |
same glibc-malloc cause (512 KB alloc per iteration) | mimalloc as
global allocator |

Left alone: `compact_sliced[(4096, 90)]` (single sighting) and the CUDA
walltime benches (hosted-runner walltime noise, a runner config issue).

The allocator swap shifts every benchmark in the two touched binaries
once — see the comment below. Needs a one-time CodSpeed acknowledgment,
like #8742.

## What changes are included in this PR?

One commit per benchmark; bench files only. Ran `cargo check` + `clippy`
on the three bench targets, smoke-ran the binaries, `cargo +nightly
fmt`.

---------

Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant