feat: exploit PiecewiseSequence for faster take by danking · Pull Request #8802 · vortex-data/vortex

danking · 2026-07-16T19:03:21Z

Extension pushes take down into its children. This ensures extension types benefit from the other PiecewiseSequence optimizations.
Ensure List(List(...(List(Primitive))).take(indices) keeps broadening the slices all the way down to the Primitive array.
Implement list view rebuild using PiecewiseSequence, since rebuild is just take anyway!

codspeed-hq · 2026-07-16T19:10:25Z

Merging this PR will improve performance by ×2.8

⚡ 5 improved benchmarks
✅ 1793 untouched benchmarks
⏩ 44 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Simulation	`struct_large`	431.1 µs	39.2 µs	×11
⚡	Simulation	`fsl_large`	615.6 µs	91.5 µs	×6.7
⚡	Simulation	`list_i32_large`	172.7 µs	96.6 µs	+78.86%
⚡	Simulation	`i32_large`	277.8 µs	237.7 µs	+16.87%
⚡	Simulation	`true_count_vortex_buffer[128]`	580.6 ns	522.2 ns	+11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing codex/piecewise-sequential-run-take-consumers (c8c6f81) with develop (2ea51cb)}

44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Signed-off-by: Daniel King <dan@spiraldb.com>

github-actions · 2026-07-20T19:38:54Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`c8c6f81`		1	Explore Profiling Data

Previous Runs (6)

Status	Commit	Attempt	Link
🟢 Done	`9386ccb`	1	Explore Profiling Data
🟢 Done	`15dfce8`	1	Explore Profiling Data
🟢 Done	`7cb426a`	1	Explore Profiling Data
🟢 Done	`9c642f5`	1	Explore Profiling Data
🟢 Done	`9ffb47a`	1	Explore Profiling Data
🟢 Done	`09ca199`	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-07-20T19:39:48Z

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +2.6%
Engines: DataFusion Likely regression (+12.0%, medium confidence) · DuckDB No clear signal (-6.0%, low confidence)
Vortex (geomean): 1.009x ➖
Parquet (geomean): 0.996x ➖
Shifts: Parquet (control) -0.4% · Median polish -0.4%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.083x ➖, 0↑ 1↓)

name	PR `c8c6f81` (ns)	base `2ea51cb` (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	10141231	9515901	1.07
vortex_q01/datafusion:vortex-file-compressed 🚨	6472131	5883674	1.10

datafusion / parquet (0.966x ➖, 0↑ 0↓)

name	PR `c8c6f81` (ns)	base `2ea51cb` (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	20160152	20614980	0.98
vortex_q01/datafusion:parquet	4468263	4678203	0.96

duckdb / vortex-file-compressed (0.965x ➖, 0↑ 0↓)

name	PR `c8c6f81` (ns)	base `2ea51cb` (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	9810611	9893308	0.99
vortex_q01/duckdb:vortex-file-compressed	5725403	6094115	0.94

duckdb / parquet (1.027x ➖, 0↑ 0↓)

name	PR `c8c6f81` (ns)	base `2ea51cb` (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	24078268	23323200	1.03
vortex_q01/duckdb:parquet	9544633	9341507	1.02

No file size changes detected.

Signed-off-by: Daniel King <dan@spiraldb.com>

danking · 2026-07-20T19:34:03Z

        indices: &ArrayRef,
        ctx: &mut ExecutionCtx,
    ) -> VortexResult<Option<ArrayRef>> {
+        if let Some(piecewise_indices) = indices.as_opt::<PiecewiseSequence>()


To be clear, this would happen because we have a List(List) and we've taken the outer list. This ensures we doubly explode those outer indices.

danking · 2026-07-20T19:41:51Z

@@ -127,6 +145,438 @@ fn _take<I: IntegerPType, O: IntegerPType, OutputOffsetType: IntegerPType>(
    .into_array())
 }



This whole sequence kinda sucks but I don't know how to make it less verbose. We need to dispatch on four things: starts, lengths, input-list-offsets, and output-list-offsets.

And everything is splayed in two because there's the constant vs non-constant lengths path.

I think you can come up with a widening that would reduce it for temporary state but not quite sure otoh. We can merge this and figure out how to abbreviate it

Signed-off-by: Daniel King <dan@spiraldb.com>

danking force-pushed the codex/piecewise-sequential-take branch from 01e95cb to eb83451 Compare July 16, 2026 19:19

danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 4272937 to 02adf44 Compare July 16, 2026 19:23

danking changed the title ~~Port PiecewiseSequential run-take consumers~~ Port PiecewiseSequence run-take consumers Jul 16, 2026

danking force-pushed the codex/piecewise-sequential-take branch from eb83451 to e79ec4b Compare July 16, 2026 20:29

danking force-pushed the codex/piecewise-sequential-run-take-consumers branch 2 times, most recently from 4ff991e to de4a71f Compare July 16, 2026 20:36

danking force-pushed the codex/piecewise-sequential-take branch from b615895 to 6dcc52a Compare July 16, 2026 23:41

danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 12a04f6 to 6280163 Compare July 16, 2026 23:52

danking added the changelog/performance A performance improvement label Jul 16, 2026

danking force-pushed the codex/piecewise-sequential-take branch from 6dcc52a to a2cbdf0 Compare July 17, 2026 00:15

danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 6280163 to 71c3d0e Compare July 17, 2026 00:16

danking force-pushed the codex/piecewise-sequential-take branch from a2cbdf0 to d694338 Compare July 17, 2026 00:17

danking force-pushed the codex/piecewise-sequential-run-take-consumers branch 12 times, most recently from 490e9db to ed7bbe2 Compare July 20, 2026 16:11

danking force-pushed the codex/piecewise-sequential-take branch from c94c4ef to 1fdda1c Compare July 20, 2026 16:11

danking force-pushed the codex/piecewise-sequential-run-take-consumers branch 3 times, most recently from 56b0394 to 80bfa43 Compare July 20, 2026 18:10

Base automatically changed from codex/piecewise-sequential-take to develop July 20, 2026 19:20

danking added 11 commits July 20, 2026 15:22

Port PiecewiseSequence run take consumers

884542d

Signed-off-by: Daniel King <dan@spiraldb.com>

Use unit multipliers for PiecewiseSequence consumers

5027bff

Signed-off-by: Daniel King <dan@spiraldb.com>

Specialize constant PiecewiseSequence runs

68a7eb5

Signed-off-by: Daniel King <dan@spiraldb.com>

Inline List PiecewiseSequence range length checks

b69e2db

Signed-off-by: Daniel King <dan@spiraldb.com>

Cover null ListView rebuild placeholders

90e5007

Signed-off-by: Daniel King <dan@spiraldb.com>

Mask null ListView take metadata

e2df924

Signed-off-by: Daniel King <dan@spiraldb.com>

Rename List contiguous slice helper uses

4e4650f

Signed-off-by: Daniel King <dan@spiraldb.com>

Remove redundant ListView validity clone

daa506e

Signed-off-by: Daniel King <dan@spiraldb.com>

Simplify zero-length list element sizing

e69c305

Signed-off-by: Daniel King <dan@spiraldb.com>

Use Columnar lengths in List take

f1880b1

Signed-off-by: Daniel King <dan@spiraldb.com>

Rely on PiecewiseSequence validation in List take

09ca199

Signed-off-by: Daniel King <dan@spiraldb.com>

danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 80bfa43 to 09ca199 Compare July 20, 2026 19:24

Rename List take slice dispatch helpers

9ffb47a

Signed-off-by: Daniel King <dan@spiraldb.com>

danking changed the title ~~Port PiecewiseSequence run-take consumers~~ feat: exploit PiecewiseSequence for faster take Jul 20, 2026

danking added 5 commits July 20, 2026 15:57

Preserve ListView take metadata

9c642f5

Signed-off-by: Daniel King <dan@spiraldb.com>

Use valid null ListView ranges in rebuild test

7cb426a

Signed-off-by: Daniel King <dan@spiraldb.com>

Restore ListView take implementation

2ef9e09

Signed-off-by: Daniel King <dan@spiraldb.com>

Keep ListView rebuild helper name

15dfce8

Signed-off-by: Daniel King <dan@spiraldb.com>

Handle nullable lists in PiecewiseSequence take

9386ccb

Signed-off-by: Daniel King <dan@spiraldb.com>

danking commented Jul 20, 2026

View reviewed changes

danking marked this pull request as ready for review July 20, 2026 21:00

danking requested review from joseph-isaacs and robert3005 July 20, 2026 21:01

robert3005 approved these changes Jul 20, 2026

View reviewed changes

danking enabled auto-merge (squash) July 20, 2026 21:44

Fix list PiecewiseSequence clippy lint

c8c6f81

Signed-off-by: Daniel King <dan@spiraldb.com>

danking merged commit add6944 into develop Jul 20, 2026
74 checks passed

danking deleted the codex/piecewise-sequential-run-take-consumers branch July 20, 2026 22:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: exploit PiecewiseSequence for faster take#8802

feat: exploit PiecewiseSequence for faster take#8802
danking merged 18 commits into
developfrom
codex/piecewise-sequential-run-take-consumers

danking commented Jul 16, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

danking Jul 20, 2026

Uh oh!

danking Jul 20, 2026

Uh oh!

danking Jul 20, 2026

Uh oh!

robert3005 Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -127,6 +145,438 @@ fn _take<I: IntegerPType, O: IntegerPType, OutputOffsetType: IntegerPType>(
		.into_array())
		}

Uh oh!

Conversation

danking commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by ×2.8

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Vortex queries 📖

Uh oh!

danking Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

danking Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

danking Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

danking commented Jul 16, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading