Skip to content

feat(onpair): promote OnPair to a stable read-only encoding#8872

Merged
joseph-isaacs merged 4 commits into
developfrom
ji/onpair-non-expiremental
Jul 21, 2026
Merged

feat(onpair): promote OnPair to a stable read-only encoding#8872
joseph-isaacs merged 4 commits into
developfrom
ji/onpair-non-expiremental

Conversation

@gargiulofrancesco

Copy link
Copy Markdown
Contributor

Rationale for this change

OnPair has been living under encodings/experimental/ while it stabilized. We now
want OnPair-encoded files to be readable by default, without yet committing to writing
them. This PR promotes OnPair to a stable encoding but enables it on the read path
only
— the compressor/writer stays gated behind the unstable_encodings feature so
we can turn on writing in a follow-up.

What changes are included in this PR?

  • Move the crate from encodings/experimental/onpairencodings/onpair (100% file
    renames) and drop it from the "Experimental encodings" group in the workspace
    Cargo.toml.
  • Register the OnPair reader unconditionally in register_default_encodings, so any
    session can decode OnPair arrays and push down filters against them.
  • Keep the writer/compressor disabled by default:
    • ALLOWED_ENCODINGS (file write strategy) only inserts OnPair under
      #[cfg(feature = "unstable_encodings")].
    • btrblocks ALL_SCHEMES and the onpair scheme module are gated behind
      unstable_encodings, so the default compressor never selects OnPair.
  • Tests:
    • New default_writer_does_not_allow_onpair asserts OnPair is absent from the default
      write allowlist; the reader-registration test asserts OnPair is registered with a
      Filter execute-parent kernel.
    • btrblocks scheme-selection tests that exercise writing are gated behind
      unstable_encodings and renamed to make the scope explicit
      (test_unstable_all_schemes_includes_onpair,
      test_unstable_default_btrblocks_compressor_selects_onpair).

Verified locally:

  • Default (reader-only) config: vortex-onpair, vortex-btrblocks, vortex-file
    169 tests passed.
  • unstable_encodings (writer-enabled) config: 176 tests passed (the extra 7 are the
    feature-gated write-path tests).
  • cargo clippy --all-targets --all-features (affected crates) clean;
    cargo +nightly fmt --all clean; git diff --check clean.

What APIs are changed? Are there any user-facing changes?

  • The vortex-onpair crate moves to encodings/onpair (path only; crate name and
    version are unchanged).
  • User-facing behavior change: OnPair-encoded files are now readable by default.
    Writing OnPair still requires the unstable_encodings feature — no default write
    behavior changes.

joseph-isaacs and others added 2 commits July 20, 2026 10:47
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Move encodings/experimental/onpair -> encodings/onpair and register the
OnPair reader unconditionally. The writer stays gated behind the
`unstable_encodings` feature (file ALLOWED_ENCODINGS and btrblocks
ALL_SCHEMES/scheme module), so files with OnPair can be read but not yet
written by default.

Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gargiulofrancesco gargiulofrancesco added the changelog/feature A new feature label Jul 21, 2026
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done ee01ecc 1 Explore Profiling Data
Previous Runs (2)
Status Commit Job Attempt Link
🟢 Done b061163 1 Explore Profiling Data
🟢 Done fc8ced8 1 Explore Profiling Data

Powered by Polar Signals Cloud

@codspeed-hq

codspeed-hq Bot commented Jul 21, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 1796 untouched benchmarks
⏩ 46 skipped benchmarks1


Comparing ji/onpair-non-expiremental (ee01ecc) with develop (5746d55)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +2.0%
Engines: DataFusion No clear signal (+7.1%, low confidence) · DuckDB No clear signal (-2.9%, low confidence)
Vortex (geomean): 1.009x ➖
Parquet (geomean): 0.999x ➖
Shifts: Parquet (control) -0.1% · Median polish -0.1%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.067x ➖, 0↑ 0↓)
name PR ee01ecc (ns) base 5746d55 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 10019152 9410240 1.06
vortex_q01/datafusion:vortex-file-compressed 6431205 6017158 1.07
datafusion / parquet (0.996x ➖, 0↑ 0↓)
name PR ee01ecc (ns) base 5746d55 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20682192 21234103 0.97
vortex_q01/datafusion:parquet 4748623 4660522 1.02
duckdb / vortex-file-compressed (0.973x ➖, 0↑ 0↓)
name PR ee01ecc (ns) base 5746d55 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10435104 10242605 1.02
vortex_q01/duckdb:vortex-file-compressed 5987331 6436835 0.93
duckdb / parquet (1.002x ➖, 0↑ 0↓)
name PR ee01ecc (ns) base 5746d55 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23472627 23381736 1.00
vortex_q01/duckdb:parquet 9443826 9440194 1.00

No file size changes detected.

@gargiulofrancesco
gargiulofrancesco marked this pull request as ready for review July 21, 2026 10:00
@joseph-isaacs
joseph-isaacs merged commit e767a30 into develop Jul 21, 2026
115 of 117 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/onpair-non-expiremental branch July 21, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants