Skip to content

feat(vortex-file): gate the file writer behind editions#8877

Draft
joseph-isaacs wants to merge 2 commits into
developfrom
claude/writer-editions-gating-8qy1fv
Draft

feat(vortex-file): gate the file writer behind editions#8877
joseph-isaacs wants to merge 2 commits into
developfrom
claude/writer-editions-gating-8qy1fv

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Follow-up to the editions implementation (#8871): the edition declarations existed but nothing enforced them. This PR gates the file writer behind editions. The session is the source of truth: it stores the set of enabled editions alongside the declarations, and exposes the resolved set of all enabled encodings. The default session enables core2026.07.0 (EditionId::new("core", 2026, 7, 0)), so by default:

  • the compressor only chooses schemes whose output encodings belong to the enabled editions, and
  • the set that validates every encoding serialized to disk (failing the write on a violation) is that same edition's encoding set.

What changes are included in this PR?

  • vortex-edition: EditionSession now stores the enabled editions next to the declarations: enable() (must be declared; at most one edition per family), enabled(), and enabled_encodings() — the resolved, sorted union of the enabled editions' encoding sets (None when nothing is enabled). The first-party declarations stay in the vortex facade; this crate remains types + session variable + test harness.
  • vortex (facade): register_default_editions now also enables DEFAULT_WRITE_EDITIONS (core2026.07.0, plus the newest unstable draft when the unstable_encodings feature opts in) on the session. New tests: a golden pin of the frozen core2026.07.0 encoding set, per-edition harness validation, subset properties, and a check that every default compression scheme's produced encodings are covered by the default write editions.
  • vortex-compressor: new required Scheme::produced_encodings(&self) -> Vec<ArrayId> trait method declaring the array encodings a scheme may emit (empty = only canonical arrays; cascaded children are covered by their own schemes). CascadingCompressor::with_enabled_editions_gating makes scheme selection resolve the session's enabled encodings from the execution context at compression time and skip schemes whose outputs are not all allowed; sessions with no enabled editions are not gated. Unit-tested with a synthetic edition/scheme.
  • vortex-btrblocks: every scheme declares its produced encodings (including the env-gated Patched output of BitPacking/ALP). BtrBlocksCompressorBuilder gains gate_by_enabled_editions() (session gating pass-through) and retain_allowed_encodings() (static filtering for explicit allow sets).
  • vortex-layout: FlatLayoutStrategy::with_enabled_editions_gating resolves the allowed encodings from the session's enabled editions at write_stream time — an encoding outside them fails the write via the existing normalize-with-Error validation — falling back to the explicit allowed_encodings set when the session enables no editions.
  • vortex-file: the default WriteStrategyBuilder wires all of this up: flat writer gated by enabled editions with the static ALLOWED_ENCODINGS registry as fallback, and both the data and stats compressors gated at compression time. An explicit with_allow_encodings set overrides the session gate everywhere (unchanged behavior for vortex-jni / vortex-parquet-variant). The writer's deterministic pre-populated ArrayContext likewise comes from the session's enabled encodings. Experimental patches remain a runtime opt-out: Patched is only writable when the experimental environment variable is set, mirroring when compression produces it.

Checks run (after rebasing onto develop at #8871): cargo nextest run -p vortex-edition -p vortex-compressor -p vortex-btrblocks -p vortex-layout -p vortex-file -p vortex (411 passed; also with --all-features for file/btrblocks/compressor), cargo test --doc for the touched crates, cargo +nightly fmt --all, and cargo clippy --all-targets --all-features for the touched crates plus vortex-tensor, vortex-jni, and vortex-parquet-variant (clean). Workspace-wide clippy could not be run: vortex-duckdb's build script cannot download DuckDB sources in this sandbox.

What APIs are changed? Are there any user-facing changes?

  • Scheme has a new required method produced_encodings() -> Vec<ArrayId> — a breaking change for third-party schemes, which must now declare their outputs.
  • EditionSession::{enable, enabled, enabled_encodings}, CascadingCompressor::with_enabled_editions_gating, BtrBlocksCompressorBuilder::{gate_by_enabled_editions, retain_allowed_encodings}, FlatLayoutStrategy::with_enabled_editions_gating, and vortex::editions::DEFAULT_WRITE_EDITIONS are new.
  • Behavior change for sessions built via the vortex facade: encodings outside the enabled editions (fastlanes.delta, vortex.onpair, vortex.zstd_buffers without the unstable_encodings feature, vortex.patched without the env opt-in) are no longer writable — writes containing them fail at the flat-layout validation instead of producing files without the read-forever guarantee. Sessions that enable no editions keep the previous static ALLOWED_ENCODINGS behavior.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2

@joseph-isaacs
joseph-isaacs marked this pull request as draft July 21, 2026 12:41
Base automatically changed from claude/editions-impl-fiicrg to develop July 21, 2026 13:47
The session is the source of truth for what a writer may emit:
EditionSession now stores the set of enabled editions alongside the
declarations — enable() (validated, at most one edition per family),
enabled(), and enabled_encodings(), the resolved union of the enabled
editions' encoding sets. register_default_editions declares the
editions and enables the default write editions: core2026.07.0, plus
the newest unstable draft under the `unstable_encodings` feature.

The writer resolves its allow set from the session at write time:

- The flat leaf writer's normalize-with-error validation fails the
  write for any encoding outside the session's enabled editions.
- The writer's pre-populated ArrayContext is seeded from the same set.
- Compression is gated at scheme-selection time: the new required
  Scheme::produced_encodings method declares the array encodings a
  scheme may emit (empty means only canonical arrays), and
  CascadingCompressor::with_enabled_editions_gating skips schemes
  whose outputs are not all within the enabled editions, resolved
  from the execution context per compression.

Sessions with no enabled editions fall back to the static
ALLOWED_ENCODINGS registry, and explicit with_allow_encodings sets
still override everything. Experimental patches remain a runtime
opt-out: Patched is only writable when the experimental environment
variable is set, mirroring when compression produces it.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2
@joseph-isaacs
joseph-isaacs force-pushed the claude/writer-editions-gating-8qy1fv branch from bf7ca36 to 2da25df Compare July 21, 2026 13:54
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done c84896c 1 Explore Profiling Data
Previous Runs (1)
Status Commit Job Attempt Link
🟢 Done 2da25df 1 Explore Profiling Data

Powered by Polar Signals Cloud

@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Jul 21, 2026 — with Claude
- Qualify the DEFAULT_WRITE_EDITIONS intra-doc link in the editions
  module docs: links in the module's inner docs resolve at the facade's
  `pub mod` site, so unqualified links break rustdoc under -D warnings.
  Also move the stray "Logical Vortex data types." doc line back onto
  the dtype module.
- Update the file-size golden values in the Python write-options
  doctests: the writer's pre-populated ArrayContext is now seeded from
  the session's enabled editions, which changes the footer's encoding
  list and shrinks the written files by 40 bytes.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +9.7%
Engines: DataFusion No clear signal (+7.3%, environment too noisy confidence) · DuckDB Likely regression (+12.2%, high confidence)
Vortex (geomean): 1.092x ➖
Parquet (geomean): 1.042x ➖
Shifts: Parquet (control) +4.2% · Median polish +8.8%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.127x ❌, 0↑ 2↓)
name PR c84896c (ns) base 94e593e (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 🚨 10735130 9541200 1.13
vortex_q01/datafusion:vortex-file-compressed 🚨 6741768 5974462 1.13
datafusion / parquet (1.050x ➖, 0↑ 0↓)
name PR c84896c (ns) base 94e593e (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 21479881 21214230 1.01
vortex_q01/datafusion:parquet 5161545 4736768 1.09
duckdb / vortex-file-compressed (1.160x ❌, 0↑ 2↓)
name PR c84896c (ns) base 94e593e (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 🚨 11416035 9885437 1.15
vortex_q01/duckdb:vortex-file-compressed 🚨 7028035 6028973 1.17
duckdb / parquet (1.034x ➖, 0↑ 0↓)
name PR c84896c (ns) base 94e593e (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 24060989 23250798 1.03
vortex_q01/duckdb:parquet 9689434 9373387 1.03

File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
File Scale Format Base HEAD Change %
test.vortex 1.0 vortex-file-compressed 80.59 MB 80.59 MB +248 B +0.0%

Totals:

  • vortex-file-compressed: 80.85 MB → 80.85 MB (+0.0%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants