docs: introduce Vortex editions for users of the file format by joseph-isaacs · Pull Request #8855 · vortex-data/vortex

joseph-isaacs · 2026-07-20T13:56:22Z

Rationale for this change

Vortex needs a named, user-visible compatibility contract for which encodings a writer may put in a file. This PR introduces editions: a named grouping of encodings that records exactly which encodings a Vortex file may contain and when each encoding joined the set. Writing under an edition guarantees every future Vortex release can read — and execute queries over — the file. The concept fills the forward/backward-compatibility seam already noted in register_default_encodings ("different Vortex 'Editions' that may support different sets of encodings") and gives unknown-encoding errors a stable docs page to link to. A follow-up PR adds the Rust implementation (vortex-edition crate) and the generated registry.

github-actions · 2026-07-20T13:59:15Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`441b383`		1	Explore Profiling Data

Previous Runs (1)

Status	Commit	Job	Attempt	Link
🟢 Done	`fd9a9e6`		1	Explore Profiling Data

Powered by Polar Signals Cloud

An edition names the exact set of encodings a writer may put in a file, giving every file written under it a forever read-and-execute guarantee. This spec page explains the model for users of Vortex: editions constrain writers while readers are cumulative, family-namespaced editions (core2026.07.0) compose additively, writer presets choose within an edition without affecting portability, and unknown-encoding errors link back to the registry. Defines the first edition core2026.07.0 (the 32 encodings the default writer emits today) and the core2026.10.0 draft. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KdR42Svu74NcNzC7XJYwLr

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

github-actions · 2026-07-20T14:00:12Z

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +4.5%
Engines: DataFusion No clear signal (+2.9%, low confidence) · DuckDB No clear signal (+6.1%, low confidence)
Vortex (geomean): 1.052x ➖
Parquet (geomean): 1.029x ➖
Shifts: Parquet (control) +2.9% · Median polish +5.2%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.067x ➖, 0↑ 0↓)

name	PR `441b383` (ns)	base `b7b01d3` (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	10498063	9831831	1.07
vortex_q01/datafusion:vortex-file-compressed	6622469	6216077	1.07

datafusion / parquet (1.037x ➖, 0↑ 0↓)

name	PR `441b383` (ns)	base `b7b01d3` (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	21647562	21543706	1.00
vortex_q01/datafusion:parquet	5041710	4713514	1.07

duckdb / vortex-file-compressed (1.083x ➖, 0↑ 0↓)

name	PR `441b383` (ns)	base `b7b01d3` (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	10764440	9900408	1.09
vortex_q01/duckdb:vortex-file-compressed	6440336	5965326	1.08

duckdb / parquet (1.022x ➖, 0↑ 0↓)

name	PR `441b383` (ns)	base `b7b01d3` (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23988314	23497829	1.02
vortex_q01/duckdb:parquet	9617721	9407972	1.02

No file size changes detected.

codspeed-hq · 2026-07-20T14:00:42Z

Merging this PR will not alter performance

✅ 1670 untouched benchmarks
⏩ 44 skipped benchmarks¹

_{Comparing claude/editions-design-doc-fiicrg (441b383) with develop (b7b01d3)}

44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 · 2026-07-20T14:12:36Z

+The edition `core2026.07.0` (coming soon) is the first such edition containing all encodings currently
+enabled by the writer.


Let's enumerate these. See

vortex/vortex-btrblocks/src/builder.rs

Lines 24 to 67 in b7b01d3

pub const ALL_SCHEMES: &[&dyn Scheme] = &[

////////////////////////////////////////////////////////////////////////////////////////////////

// Integer schemes.

////////////////////////////////////////////////////////////////////////////////////////////////

// NOTE: FoR must precede BitPacking to avoid unnecessary patches.

&integer::FoRScheme,

// NOTE: ZigZag should precede BitPacking because we don't want negative numbers.

&integer::ZigZagScheme,

&integer::BitPackingScheme,

&integer::SparseScheme,

&integer::IntDictScheme,

&integer::RunEndScheme,

&integer::SequenceScheme,

&integer::IntRLEScheme,

// Prefer all other schemes above delta, for now (since its slower to decompress).

#[cfg(feature = "unstable_encodings")]

&integer::DeltaScheme::new(1.25),

////////////////////////////////////////////////////////////////////////////////////////////////

// Float schemes.

////////////////////////////////////////////////////////////////////////////////////////////////

&float::ALPScheme,

&float::ALPRDScheme,

&float::FloatDictScheme,

&float::NullDominatedSparseScheme,

&float::FloatRLEScheme,

////////////////////////////////////////////////////////////////////////////////////////////////

// String schemes.

////////////////////////////////////////////////////////////////////////////////////////////////

&string::StringDictScheme,

// Both string-fragmentation schemes are registered; the sample-based

// selector keeps whichever is smaller per column.

&string::FSSTScheme,

#[cfg(feature = "unstable_encodings")]

&string::OnPairScheme,

&string::NullDominatedSparseScheme,

////////////////////////////////////////////////////////////////////////////////////////////////

// Binary schemes.

////////////////////////////////////////////////////////////////////////////////////////////////

&binary::BinaryDictScheme,

// Decimal schemes.

&decimal::DecimalScheme,

// Temporal schemes.

&temporal::TemporalScheme,

];

I will add a rust definition that will code gen docs and gate the reader.

robert3005 · 2026-07-20T16:20:40Z

+version of vortex.
+**Editions** are used to keep track of these encodings and talk about groups of encodings.
+
+The edition `core2026.07.0` (coming soon) is the first such edition containing all encodings currently


I would drop core, not sure what this is meant to mean

robert3005 · 2026-07-20T16:20:49Z

+enabled by the writer.
+Editions are additive so an edition that comes after a previous one contains all the encodings from the previous one
+and more.
+The writer can be configured with a set of different editions (e.g. `core2026.07.0` and `unstable2026.05.0` all stable


joseph-isaacs added the changelog/docs A docs change label Jul 20, 2026

joseph-isaacs requested a review from gatesn July 20, 2026 13:57

claude and others added 2 commits July 20, 2026 13:59

update

441b383

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

joseph-isaacs force-pushed the claude/editions-design-doc-fiicrg branch from fd9a9e6 to 441b383 Compare July 20, 2026 14:00

connortsui20 reviewed Jul 20, 2026

View reviewed changes

robert3005 reviewed Jul 20, 2026

View reviewed changes

robert3005 approved these changes Jul 20, 2026

View reviewed changes

joseph-isaacs merged commit 05c7c2f into develop Jul 20, 2026
101 checks passed

joseph-isaacs deleted the claude/editions-design-doc-fiicrg branch July 20, 2026 16:25

This was referenced Jul 21, 2026

feat[edition]: session-registered editions defitions #8871

Merged

feat(vortex-file): gate the file writer behind editions #8877

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs: introduce Vortex editions for users of the file format#8855

docs: introduce Vortex editions for users of the file format#8855
joseph-isaacs merged 2 commits into
developfrom
claude/editions-design-doc-fiicrg

joseph-isaacs commented Jul 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

connortsui20 Jul 20, 2026

Uh oh!

joseph-isaacs Jul 20, 2026

Uh oh!

robert3005 Jul 20, 2026

Uh oh!

robert3005 Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		The edition `core2026.07.0` (coming soon) is the first such edition containing all encodings currently
		enabled by the writer.

	pub const ALL_SCHEMES: &[&dyn Scheme] = &[
	////////////////////////////////////////////////////////////////////////////////////////////////
	// Integer schemes.
	////////////////////////////////////////////////////////////////////////////////////////////////
	// NOTE: FoR must precede BitPacking to avoid unnecessary patches.
	&integer::FoRScheme,
	// NOTE: ZigZag should precede BitPacking because we don't want negative numbers.
	&integer::ZigZagScheme,
	&integer::BitPackingScheme,
	&integer::SparseScheme,
	&integer::IntDictScheme,
	&integer::RunEndScheme,
	&integer::SequenceScheme,
	&integer::IntRLEScheme,
	// Prefer all other schemes above delta, for now (since its slower to decompress).
	#[cfg(feature = "unstable_encodings")]
	&integer::DeltaScheme::new(1.25),
	////////////////////////////////////////////////////////////////////////////////////////////////
	// Float schemes.
	////////////////////////////////////////////////////////////////////////////////////////////////
	&float::ALPScheme,
	&float::ALPRDScheme,
	&float::FloatDictScheme,
	&float::NullDominatedSparseScheme,
	&float::FloatRLEScheme,
	////////////////////////////////////////////////////////////////////////////////////////////////
	// String schemes.
	////////////////////////////////////////////////////////////////////////////////////////////////
	&string::StringDictScheme,
	// Both string-fragmentation schemes are registered; the sample-based
	// selector keeps whichever is smaller per column.
	&string::FSSTScheme,
	#[cfg(feature = "unstable_encodings")]
	&string::OnPairScheme,
	&string::NullDominatedSparseScheme,
	////////////////////////////////////////////////////////////////////////////////////////////////
	// Binary schemes.
	////////////////////////////////////////////////////////////////////////////////////////////////
	&binary::BinaryDictScheme,
	// Decimal schemes.
	&decimal::DecimalScheme,
	// Temporal schemes.
	&temporal::TemporalScheme,
	];

Uh oh!

Conversation

joseph-isaacs commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Vortex queries 📖

Uh oh!

codspeed-hq Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

connortsui20 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

joseph-isaacs Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joseph-isaacs commented Jul 20, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 20, 2026 •

edited

Loading