Unify primitive and decimal physical storage

[connortsui20]

Summary

• introduce shared FixedWidthData / FixedWidthType storage for canonical Primitive and Decimal arrays
• retain distinct Primitive and Decimal VTables, logical behavior, matchers, and existing array IDs
• preserve the existing Primitive and Decimal serialized metadata and buffer/validity layouts
• update Decimal call sites to use its distinct canonical representation

Compatibility

The existing vortex.primitive and vortex.decimal IDs and their serialized layouts remain unchanged. The historical compatibility suite passed all 455 published fixtures across Vortex 0.64.0 and 0.68.0 through 0.79.0 on the combined implementation.

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	f124538		1	Explore Profiling Data

Previous Runs (5)

Status	Commit	Job	Attempt	Link
🟢 Done	2880b5f		1	Explore Profiling Data
🟢 Done	35a4157		1	Explore Profiling Data
🟢 Done	2e0b1e2		1	Explore Profiling Data
🟢 Done	f00395e		1	Explore Profiling Data
🟢 Done	3ba3236		1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[codecov]

Codecov Report

❌ Patch coverage is 73.25228% with 176 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.03%. Comparing base (2ea51cb) to head (f00395e).

Files with missing lines	Patch %	Lines
vortex-array/src/arrays/decimal/array.rs	64.56%	45 Missing ⚠️
vortex-array/src/arrays/decimal/compute/cast.rs	80.58%	20 Missing ⚠️
vortex-array/src/arrays/fixed_width.rs	81.18%	19 Missing ⚠️
vortex-array/src/arrays/decimal/vtable/mod.rs	62.79%	16 Missing ⚠️
vortex-cuda/src/arrow/canonical.rs	0.00%	16 Missing ⚠️
vortex-cuda/src/kernel/arrays/dict.rs	0.00%	10 Missing ⚠️
vortex-array/src/arrays/primitive/vtable/mod.rs	73.91%	6 Missing ⚠️
vortex-array/src/arrays/decimal/compute/mask.rs	58.33%	5 Missing ⚠️
vortex-array/src/arrays/decimal/compute/rules.rs	61.53%	5 Missing ⚠️
vortex-array/src/arrays/decimal/compute/take.rs	83.33%	4 Missing ⚠️
... and 15 more

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: -1.3%
Engines: DataFusion No clear signal (+6.7%, low confidence) · DuckDB No clear signal (-8.7%, medium confidence)
Vortex (geomean): 0.900x ✅
Parquet (geomean): 0.906x ➖
Shifts: Parquet (control) -9.4% · Median polish -10.6%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

datafusion / vortex-file-compressed (0.911x ➖, 0↑ 0↓)

name	PR f124538 (ns)	base 5b688a9 (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	9702322	10644295	0.91
vortex_q01/datafusion:vortex-file-compressed	6221822	6830634	0.91

datafusion / parquet (0.854x ✅, 2↑ 0↓)

name	PR f124538 (ns)	base 5b688a9 (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet 🚀	19752361	22843824	0.86
vortex_q01/datafusion:parquet 🚀	4587277	5437392	0.84

duckdb / vortex-file-compressed (0.877x ✅, 1↑ 0↓)

name	PR f124538 (ns)	base 5b688a9 (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	9949442	10947785	0.91
vortex_q01/duckdb:vortex-file-compressed 🚀	5886171	6952563	0.85

duckdb / parquet (0.961x ➖, 0↑ 0↓)

name	PR f124538 (ns)	base 5b688a9 (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23325536	24229204	0.96
vortex_q01/duckdb:parquet	9345135	9746132	0.96

---

No file size changes detected.

[codspeed-hq]

Merging this PR will improve performance by 72.44%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 4 improved benchmarks
✅ 1835 untouched benchmarks
⏩ 3 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	varbinview_large	411 µs	164.7 µs	×2.5
⚡	Simulation	copy_nullable[65536]	1,362.6 µs	579.4 µs	×2.4
⚡	Simulation	copy_non_nullable[65536]	1,094.7 µs	799.7 µs	+36.88%
⚡	Simulation	fsl_large	99.6 µs	90.5 µs	+10.06%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/fixed-width-primitive-decimal (f124538) with develop (5b688a9)}

Footnotes

1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩