Skip to content

Add UnionArray compute functions#8884

Draft
connortsui20 wants to merge 1 commit into
ct/union-arrayfrom
ct/union-array-compute
Draft

Add UnionArray compute functions#8884
connortsui20 wants to merge 1 commit into
ct/union-arrayfrom
ct/union-array-compute

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 21, 2026

Copy link
Copy Markdown
Member

Rationale for this change

Depends on #8834. Tracking issue: #8769.

This is the deferred compute layer for UnionArray and remains a draft for later review.

What changes are included in this PR?

  • Adds UnionArray filter, take, slice, and mask execution.
  • Adds constant and chunked canonicalization.
  • Adds Union casting, uncompressed-size accounting, and recursive compression.
  • Adds focused tests for structural operations and canonicalization.

Nullable-index take remains explicitly unsupported and needs resolution before this layer is ready.

What APIs are changed? Are there any user-facing changes?

No additional public array representation APIs beyond #8834. This layer wires UnionArray into existing compute paths.

@connortsui20 connortsui20 changed the title ct/union array compute Add UnionArray compute functions Jul 21, 2026
@connortsui20 connortsui20 mentioned this pull request Jul 21, 2026
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done ab33ec0 1 Explore Profiling Data
Previous Runs (6)
Status Commit Job Attempt Link
🟢 Done f2427ce 1 Explore Profiling Data
🟢 Done e32db9e 1 Explore Profiling Data
🟢 Done 98f8f06 1 Explore Profiling Data
🟢 Done 1e867dd 1 Explore Profiling Data
🟢 Done a6b7e6c 1 Explore Profiling Data
🟢 Done 6fe53ba 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +4.5%
Engines: DataFusion No clear signal (+8.1%, low confidence) · DuckDB No clear signal (+1.0%, low confidence)
Vortex (geomean): 1.006x ➖
Parquet (geomean): 0.984x ➖
Shifts: Parquet (control) -1.6% · Median polish +1.2%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.040x ➖, 0↑ 0↓)
name PR ab33ec0 (ns) base 69f5bc5 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 10068833 9640865 1.04
vortex_q01/datafusion:vortex-file-compressed 6306581 6087569 1.04
datafusion / parquet (0.962x ➖, 0↑ 0↓)
name PR ab33ec0 (ns) base 69f5bc5 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 19895977 20804028 0.96
vortex_q01/datafusion:parquet 4487039 4638739 0.97
duckdb / vortex-file-compressed (1.017x ➖, 0↑ 0↓)
name PR ab33ec0 (ns) base 69f5bc5 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10131356 9980322 1.02
vortex_q01/duckdb:vortex-file-compressed 6014631 5899636 1.02
duckdb / parquet (1.007x ➖, 0↑ 0↓)
name PR ab33ec0 (ns) base 69f5bc5 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23421945 23225257 1.01
vortex_q01/duckdb:parquet 9419267 9364034 1.01

No file size changes detected.

@connortsui20
connortsui20 force-pushed the ct/union-array-compute branch from 6fe53ba to a6b7e6c Compare July 21, 2026 17:48
@connortsui20
connortsui20 force-pushed the ct/union-array branch 2 times, most recently from 96923e7 to 3233233 Compare July 21, 2026 17:59
@connortsui20
connortsui20 force-pushed the ct/union-array-compute branch from a6b7e6c to 1e867dd Compare July 21, 2026 17:59
@codspeed-hq

codspeed-hq Bot commented Jul 21, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 14.37%

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 1844 untouched benchmarks
⏩ 46 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation copy_nullable[65536] 579.1 µs 1,033.3 µs -43.96%
Simulation copy_non_nullable[65536] 799.4 µs 913.8 µs -12.52%
Simulation decode_varbin[(1000, 8)] 34.3 µs 26.8 µs +28.06%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/union-array-compute (ab33ec0) with ct/union-array (772b738)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.09910% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.13%. Comparing base (cd92358) to head (98f8f06).

Files with missing lines Patch % Lines
...rtex-array/src/arrays/constant/vtable/canonical.rs 49.42% 44 Missing ⚠️
vortex-array/src/scalar_fn/fns/cast/mod.rs 0.00% 17 Missing ⚠️
vortex-array/src/arrays/union/tests.rs 93.59% 13 Missing ⚠️
...gregate_fn/fns/uncompressed_size_in_bytes/union.rs 0.00% 11 Missing ⚠️
vortex-array/src/arrays/masked/execute.rs 0.00% 10 Missing ⚠️
vortex-compressor/src/compressor/cascade.rs 0.00% 8 Missing ⚠️
vortex-array/src/arrays/filter/execute/mod.rs 0.00% 6 Missing ⚠️
vortex-array/src/arrays/dict/execute.rs 0.00% 3 Missing ⚠️
...aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs 0.00% 1 Missing ⚠️
...ortex-array/src/arrays/chunked/vtable/canonical.rs 97.14% 1 Missing ⚠️
... and 1 more

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@connortsui20
connortsui20 force-pushed the ct/union-array-compute branch 2 times, most recently from e32db9e to f2427ce Compare July 21, 2026 20:31
@connortsui20
connortsui20 force-pushed the ct/union-array branch 2 times, most recently from dd86118 to 772b738 Compare July 21, 2026 21:08
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/union-array-compute branch from f2427ce to ab33ec0 Compare July 21, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant