Skip to content

Avoid redundant allocations in struct layout reader#8851

Merged
joseph-isaacs merged 1 commit into
developfrom
avoid-struct-reader-allocs
Jul 20, 2026
Merged

Avoid redundant allocations in struct layout reader#8851
joseph-isaacs merged 1 commit into
developfrom
avoid-struct-reader-allocs

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Pre-size the dtype and name vectors to their final length and push the validity entry first, instead of collecting at field-count capacity and then insert(0, ..).

The previous code collected each Vec at exact field-count capacity, then insert(0, ..) forced a reallocation and an O(n) shift of every element. Both are now gone.

Checks

  • cargo build -p vortex-layout

🤖 Generated with Claude Code

Pre-size the dtype and name vectors to their final length and push the
validity entry first, instead of collecting at field-count capacity and
then inserting at index 0. This removes both the reallocation and the
O(n) element shift caused by `insert(0, ..)`.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the changelog/fix A bug fix label Jul 20, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.17%

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 44 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation true_count_vortex_buffer[128] 580.6 ns 522.2 ns +11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing avoid-struct-reader-allocs (562cc93) with develop (c5e075a)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@joseph-isaacs joseph-isaacs added changelog/performance A performance improvement and removed changelog/fix A bug fix labels Jul 20, 2026
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review July 20, 2026 13:01
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) July 20, 2026 13:01
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 562cc93 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +1.1%
Engines: DataFusion No clear signal (+6.2%, low confidence) · DuckDB No clear signal (-3.7%, low confidence)
Vortex (geomean): 1.000x ➖
Parquet (geomean): 0.995x ➖
Shifts: Parquet (control) -0.5% · Median polish -0.5%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.040x ➖, 0↑ 0↓)
name PR 562cc93 (ns) base c5e075a (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 9794691 9222307 1.06
vortex_q01/datafusion:vortex-file-compressed 6110161 6003507 1.02
datafusion / parquet (0.979x ➖, 0↑ 0↓)
name PR 562cc93 (ns) base c5e075a (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 19672299 20168593 0.98
vortex_q01/datafusion:parquet 4504808 4585156 0.98
duckdb / vortex-file-compressed (0.973x ➖, 0↑ 0↓)
name PR 562cc93 (ns) base c5e075a (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 9773956 9919645 0.99
vortex_q01/duckdb:vortex-file-compressed 5650353 5876722 0.96
duckdb / parquet (1.010x ➖, 0↑ 0↓)
name PR 562cc93 (ns) base c5e075a (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23246140 23121933 1.01
vortex_q01/duckdb:parquet 9439667 9295209 1.02

No file size changes detected.

@joseph-isaacs
joseph-isaacs merged commit b7b01d3 into develop Jul 20, 2026
88 of 90 checks passed
@joseph-isaacs
joseph-isaacs deleted the avoid-struct-reader-allocs branch July 20, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants