Skip to content

Add fixed-size binary support#8865

Draft
connortsui20 wants to merge 3 commits into
ct/fixed-width-primitive-decimalfrom
ct/fixed-width-encoding
Draft

Add fixed-size binary support#8865
connortsui20 wants to merge 3 commits into
ct/fixed-width-primitive-decimalfrom
ct/fixed-width-encoding

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 20, 2026

Copy link
Copy Markdown
Member

Stacked on #8864.

Summary

  • add the FixedSizeBinary logical dtype, scalar representation, canonical VTable, builder, and serialization support
  • reuse the shared fixed-width physical storage introduced by the base PR while keeping a distinct vortex.fixed_size_binary array ID
  • add Arrow, Python, FFI, CUDA, file, compute, fuzz, and conformance integration
  • keep FixedSizeList as a separate canonical representation

@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 14.05%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 1836 untouched benchmarks
⏩ 3 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation copy_nullable[65536] 579.4 µs 1,033.6 µs -43.94%
Simulation copy_non_nullable[65536] 799.7 µs 913.8 µs -12.49%
Simulation decompress[u8, (1000, 4)] 31 µs 23.9 µs +29.44%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/fixed-width-encoding (97c8e02) with ct/fixed-width-primitive-decimal (f124538)

Open in CodSpeed

Footnotes

  1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from dde9039 to e13e63c Compare July 20, 2026 19:26
@connortsui20
connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 3ba3236 to 76fd71a Compare July 20, 2026 19:26
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from e13e63c to 5dd799a Compare July 20, 2026 19:50
@connortsui20
connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 76fd71a to f00395e Compare July 20, 2026 19:50
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 97c8e02 1 Explore Profiling Data
Previous Runs (4)
Status Commit Job Attempt Link
🟢 Done 0592a4f 1 Explore Profiling Data
🟢 Done 8dedc68 1 Explore Profiling Data
🟢 Done 8ef7bb2 1 Explore Profiling Data
🟢 Done 5dd799a 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: -0.8%
Engines: DataFusion No clear signal (+6.8%, environment too noisy confidence) · DuckDB No clear signal (-8.0%, medium confidence)
Vortex (geomean): 0.904x ➖
Parquet (geomean): 0.907x ➖
Shifts: Parquet (control) -9.3% · Median polish -10.0%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (0.914x ➖, 0↑ 0↓)
name PR 97c8e02 (ns) base 5b688a9 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 9882028 10644295 0.93
vortex_q01/datafusion:vortex-file-compressed 6148694 6830634 0.90
datafusion / parquet (0.856x ✅, 2↑ 0↓)
name PR 97c8e02 (ns) base 5b688a9 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 🚀 20240669 22843824 0.89
vortex_q01/datafusion:parquet 🚀 4494041 5437392 0.83
duckdb / vortex-file-compressed (0.886x ✅, 1↑ 0↓)
name PR 97c8e02 (ns) base 5b688a9 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10116646 10947785 0.92
vortex_q01/duckdb:vortex-file-compressed 🚀 5902444 6952563 0.85
duckdb / parquet (0.962x ➖, 0↑ 0↓)
name PR 97c8e02 (ns) base 5b688a9 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23374418 24229204 0.96
vortex_q01/duckdb:parquet 9356408 9746132 0.96

File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
File Scale Format Base HEAD Change %
test.vortex 1.0 vortex-file-compressed 80.59 MB 80.59 MB +48 B +0.0%

Totals:

  • vortex-file-compressed: 80.85 MB → 80.85 MB (+0.0%)

@connortsui20
connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from f00395e to 35a4157 Compare July 20, 2026 21:35
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 5dd799a to 8dedc68 Compare July 20, 2026 21:35
@connortsui20
connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 35a4157 to 2e0b1e2 Compare July 20, 2026 22:05
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 8dedc68 to 8ef7bb2 Compare July 20, 2026 22:05
@connortsui20
connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 2e0b1e2 to 35a4157 Compare July 20, 2026 22:20
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 8ef7bb2 to 8dedc68 Compare July 20, 2026 22:20
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 58.04133% with 467 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.00%. Comparing base (35a4157) to head (8dedc68).

Files with missing lines Patch % Lines
vortex-array/src/arrays/fixed_size_binary/mod.rs 60.16% 194 Missing ⚠️
encodings/sparse/src/canonical.rs 0.00% 44 Missing ⚠️
vortex-array/src/builders/fixed_size_binary.rs 55.55% 44 Missing ⚠️
...rtex-array/src/arrays/constant/vtable/canonical.rs 0.00% 17 Missing ⚠️
...x-array/src/scalar_fn/fns/binary/compare/nested.rs 0.00% 17 Missing ⚠️
vortex-array/src/arrays/dict/execute.rs 0.00% 14 Missing ⚠️
vortex-array/src/canonical.rs 80.28% 14 Missing ⚠️
vortex-datafusion/src/convert/scalars.rs 33.33% 14 Missing ⚠️
vortex-array/src/arrays/masked/execute.rs 0.00% 11 Missing ⚠️
vortex-arrow/src/convert.rs 0.00% 11 Missing ⚠️
... and 26 more

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 8dedc68 to 0592a4f Compare July 20, 2026 22:33
@connortsui20
connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 35a4157 to 2880b5f Compare July 20, 2026 22:33
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 0592a4f to 97c8e02 Compare July 20, 2026 22:42
@connortsui20
connortsui20 force-pushed the ct/fixed-width-primitive-decimal branch from 2880b5f to f124538 Compare July 20, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant