Skip to content

Use PiecewiseSequence for List take elements#8833

Open
danking wants to merge 6 commits into
developfrom
codex/list-take-piecewise-elements
Open

Use PiecewiseSequence for List take elements#8833
danking wants to merge 6 commits into
developfrom
codex/list-take-piecewise-elements

Conversation

@danking

@danking danking commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

This is just a simplification. List::take already had a specialized implementation. The new PiecewiseSequence specializations in various primitive arrays supersede this change.

EDIT: Heh, I guess the PiecewiseSequence stuff is actually even faster than old List::take. Nice.


codex:

Summary

  • replace List take's dense per-element index expansion with PiecewiseSequence element runs
  • preserve FSL-style null handling: null index/source rows emit zero-length placeholder runs while output validity still comes from Validity::take
  • let the elements child choose its own optimized take implementation through the normal ArrayRef::take path

Validation

  • cargo +nightly fmt --all
  • cargo test -p vortex-array arrays::list::compute::take
  • cargo check -p vortex-array
  • cargo clippy -p vortex-array --all-targets --all-features -- -D warnings
  • git diff --check

@codspeed-hq

codspeed-hq Bot commented Jul 17, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 26.45%

⚡ 13 improved benchmarks
✅ 1785 untouched benchmarks
⏩ 44 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation copy_nullable[65536] 1.4 ms 1 ms +32.05%
Simulation take_filter_list_nullable_random_mask_random_indices[256, 50] 129.3 µs 99.4 µs +30.05%
Simulation take_filter_list_slice_mask_random_indices[768, 50] 119.7 µs 92.4 µs +29.6%
Simulation take_filter_list_nullable_random_mask_random_indices[768, 50] 133.9 µs 103.5 µs +29.39%
Simulation take_filter_list_slice_mask_random_indices[256, 50] 118.8 µs 92 µs +29.03%
Simulation take_filter_list_small_random_mask_random_indices[256, 10] 116.9 µs 90.9 µs +28.51%
Simulation take_filter_list_small_uncached_random_mask_random_indices[256, 10] 117.2 µs 91.6 µs +28.03%
Simulation take_filter_list_small_uncached_random_mask_random_indices[768, 10] 121 µs 94.9 µs +27.58%
Simulation take_filter_list_small_random_mask_random_indices[768, 10] 120.9 µs 94.9 µs +27.41%
Simulation take_filter_list_random_mask_random_indices[256, 50] 121.9 µs 96.1 µs +26.86%
Simulation take_filter_list_random_mask_random_indices[768, 50] 127.2 µs 100.5 µs +26.55%
Simulation copy_non_nullable[65536] 1,089.2 µs 908.6 µs +19.88%
Simulation take_fsl_f16_force_manual_range_copy[1, 10] 14.3 µs 12.9 µs +10.52%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing codex/list-take-piecewise-elements (e0096ca) with develop (add6944)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from c4a0358 to c76531d Compare July 17, 2026 20:05
@danking
danking force-pushed the codex/list-take-piecewise-elements branch 2 times, most recently from 77d37d8 to b7ad4df Compare July 17, 2026 20:15
@danking danking added the changelog/performance A performance improvement label Jul 17, 2026
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 80f7b89 to 30870aa Compare July 17, 2026 20:19
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from b7ad4df to f0eec2d Compare July 17, 2026 20:19
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 30870aa to e777ab4 Compare July 17, 2026 20:25
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from f0eec2d to d40cee8 Compare July 17, 2026 20:25
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from e777ab4 to e4a0543 Compare July 17, 2026 20:37
@danking
danking force-pushed the codex/list-take-piecewise-elements branch 2 times, most recently from 243ed19 to a237158 Compare July 20, 2026 16:11
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 490e9db to ed7bbe2 Compare July 20, 2026 16:11
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from a237158 to e134f5b Compare July 20, 2026 16:37
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch 2 times, most recently from efafe71 to 56b0394 Compare July 20, 2026 16:56
@danking
danking force-pushed the codex/list-take-piecewise-elements branch 2 times, most recently from ffa4184 to e88d96f Compare July 20, 2026 18:10
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 56b0394 to 80bfa43 Compare July 20, 2026 18:10
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from e88d96f to 3d3a91c Compare July 20, 2026 19:24
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 80bfa43 to 09ca199 Compare July 20, 2026 19:24
@danking
danking force-pushed the codex/list-take-piecewise-elements branch 5 times, most recently from 655fad8 to e05bf6e Compare July 20, 2026 20:17
.len()
.checked_add(1)
.ok_or_else(|| vortex_err!("List take offsets length overflow"))?;
let mut new_offsets = BufferMut::<OutputOffsetType>::with_capacity(offsets_capacity);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, just maybe this name is too long? OutOffset?

Comment thread vortex-array/src/arrays/list/compute/take.rs
Base automatically changed from codex/piecewise-sequential-run-take-consumers to develop July 20, 2026 22:00
Signed-off-by: Daniel King <dan@spiraldb.com>
danking added 5 commits July 20, 2026 18:03
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from e05bf6e to e0096ca Compare July 20, 2026 22:04
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done e0096ca 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (medium confidence)
Attributed Vortex impact: -4.8%
Engines: DataFusion No clear signal (+9.2%, low confidence) · DuckDB Likely improvement (-17.1%, high confidence)
Vortex (geomean): 0.952x ➖
Parquet (geomean): 0.976x ➖
Shifts: Parquet (control) -2.4% · Median polish -2.4%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.051x ➖, 0↑ 0↓)
name PR e0096ca (ns) base add6944 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 9786421 9338219 1.05
vortex_q01/datafusion:vortex-file-compressed 6376283 6048928 1.05
datafusion / parquet (0.962x ➖, 0↑ 0↓)
name PR e0096ca (ns) base add6944 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20312101 21059503 0.96
vortex_q01/datafusion:parquet 4562846 4750673 0.96
duckdb / vortex-file-compressed (0.821x ✅, 2↑ 0↓)
name PR e0096ca (ns) base add6944 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 🚀 9936073 12173747 0.82
vortex_q01/duckdb:vortex-file-compressed 🚀 6176471 7472232 0.83
duckdb / parquet (0.991x ➖, 0↑ 0↓)
name PR e0096ca (ns) base add6944 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23522457 23276573 1.01
vortex_q01/duckdb:parquet 9398345 9680554 0.97

No file size changes detected.

@robert3005
robert3005 marked this pull request as ready for review July 21, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants