Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ members = [
"vortex-mask",
"vortex-utils",
"vortex-session",
"vortex-edition",
"vortex-flatbuffers",
"vortex-metrics",
"vortex-io",
Expand Down Expand Up @@ -295,6 +296,7 @@ vortex-compute = { version = "0.1.0", path = "./vortex-compute", default-feature
vortex-datafusion = { version = "0.1.0", path = "./vortex-datafusion", default-features = false }
vortex-datetime-parts = { version = "0.1.0", path = "./encodings/datetime-parts", default-features = false }
vortex-decimal-byte-parts = { version = "0.1.0", path = "encodings/decimal-byte-parts", default-features = false }
vortex-edition = { version = "0.1.0", path = "./vortex-edition", default-features = false }
vortex-error = { version = "0.1.0", path = "./vortex-error", default-features = false }
vortex-fastlanes = { version = "0.1.0", path = "./encodings/fastlanes", default-features = false }
vortex-file = { version = "0.1.0", path = "./vortex-file", default-features = false }
Expand Down
12 changes: 7 additions & 5 deletions docs/specs/editions.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@ Vortex defines an evergrowing set of serializable array encodings, once written
version of vortex.
**Editions** are used to keep track of these encodings and talk about groups of encodings.

The edition `core2026.07.0` (coming soon) is the first such edition containing all encodings currently
enabled by the writer.
The first edition, `core2025.05.0`, contains the stable encodings that could be written by Vortex
`0.36.0`. This is the release from which the Vortex file format is considered stable. Later `core`
editions add stable encodings released after that compatibility boundary.
Editions are additive so an edition that comes after a previous one contains all the encodings from the previous one
and more.
The writer can be configured with a set of different editions (e.g. `core2026.07.0` and `unstable2026.05.0` all stable
encoding released before July 2026 and all unstable encodings from May 2026).
The writer can be configured with a set of different editions (for example, `core2026.07.0` and
`unstable2026.06.0` select stable encodings released through July 2026 and unstable encodings
released through June 2026).

Editions can be used to constrain your minimum required vortex reader, since latest version over vortex across all
editions is the earliest version of vortex required to read that file.
Expand Down Expand Up @@ -57,4 +59,4 @@ indefinitely, so deprecation never invalidates existing files.

## Edition registry

Coming soon..
Coming soon..
1 change: 0 additions & 1 deletion vortex-array/src/arrays/decimal/vtable/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,6 @@ impl VTable for Decimal {
dtype: &DType,
len: usize,
metadata: &[u8],

buffers: &[BufferHandle],
children: &dyn ArrayChildren,
_session: &VortexSession,
Expand Down
24 changes: 24 additions & 0 deletions vortex-edition/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[package]
name = "vortex-edition"
authors.workspace = true
description = "Definitions of Vortex editions: named, frozen sets of encodings with a read-compatibility guarantee"
edition = { workspace = true }
homepage = { workspace = true }
categories = { workspace = true }
include = { workspace = true }
keywords = { workspace = true }
license = { workspace = true }
readme = { workspace = true }
repository = { workspace = true }
rust-version = { workspace = true }
version = { workspace = true }

[package.metadata.docs.rs]
all-features = true

[lints]
workspace = true

[dependencies]
parking_lot = { workspace = true }
vortex-session = { workspace = true }
262 changes: 262 additions & 0 deletions vortex-edition/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,262 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright the Vortex contributors

//! Definitions of Vortex *editions*: named, frozen sets of encodings that a writer may put in
//! a file, carrying a forever read-compatibility guarantee.
//!
//! Editions live on the session, like encodings do: [`EditionSession`] is the session
//! variable holding the edition registry, populated at initialization time. Declarations
//! are plain constants — an [`EditionId`] plus an [`Edition`] record, and one
//! [`EditionInclusion`] per encoding stating that it is a member of an edition *and every
//! later edition of the same family*. Any crate can register declarations into a session,
//! so inclusions can live next to the encoding they describe.
//!
//! An edition is a **draft** until its [`Edition::min_vortex_version`] is recorded —
//! recording it is the act of freezing. The per-edition encoding sets are computed from the
//! registered declarations by [`EditionSession::encodings_in`], and correctness is enforced
//! by unit tests: [`EditionSession::validate`] checks a whole registry, and
//! [`test_harness::validate_edition`] validates one edition's constraints — call it once in
//! the `#[cfg(test)]` module of each edition definition.
//!
//! This crate defines only the types and the session variable; the first-party edition
//! declarations live in the public `vortex` crate (`vortex::editions`), which seeds them
//! into the default session. See the published spec at
//! <https://docs.vortex.dev/specs/editions.html>.

mod session;
pub mod test_harness;
#[cfg(test)]
mod tests;

use std::error::Error;
use std::fmt;
use std::fmt::Debug;
use std::fmt::Display;
use std::fmt::Formatter;

pub use session::EditionSession;
pub use session::EditionSessionExt;
use vortex_session::registry::Id;

/// The identifier of an edition, e.g. `core2026.07.0`.
///
/// The `family` names an independently versioned, additive group of encodings (`core` is the
/// set the default writer emits). The date components record when the edition was frozen and
/// order editions chronologically *within* a family; there is no ordering across families.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct EditionId {
/// The edition family, e.g. `core`.
pub family: &'static str,
/// Year the edition was cut.
pub year: u16,
/// Month the edition was cut.
pub month: u8,
/// Distinguishes editions cut in the same month; normally `0`.
pub version: u8,
}

impl EditionId {
/// Create an edition identifier. Validated by [`EditionId::validate`], which
/// [`test_harness::validate_edition`] exercises
/// per edition in unit tests.
pub const fn new(family: &'static str, year: u16, month: u8, version: u8) -> Self {
Self {
family,
year,
month,
version,
}
}

/// Returns true if `self` is the same edition as `other` or an earlier edition of the
/// same family. Editions of different families are never ordered.
pub fn is_at_or_before(&self, other: &EditionId) -> bool {
self.family == other.family
&& (self.year, self.month, self.version) <= (other.year, other.month, other.version)
}

/// Validate the identifier's form: a non-empty lowercase family, a four-digit year,
/// and a month in 01-12. Checked for every declared edition by
/// [`EditionSession::validate`] and per edition by
/// [`test_harness::validate_edition`].
pub fn validate(&self) -> Result<(), EditionError> {
if self.family.is_empty() || !self.family.chars().all(|c| c.is_ascii_lowercase()) {
return Err(EditionError::new(format!(
"edition {self} must have a non-empty lowercase family, e.g. `core`"
)));
}
if !(1000..=9999).contains(&self.year) {
return Err(EditionError::new(format!(
"edition {self} must have a four-digit year"
)));
}
if !(1..=12).contains(&self.month) {
return Err(EditionError::new(format!(
"edition {self} must have a month in 01-12"
)));
}
Ok(())
}
}

impl Display for EditionId {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
write!(
f,
"{}{}.{:02}.{}",
self.family, self.year, self.month, self.version
)
}
}

/// An edition: a named set of encodings with a read-compatibility guarantee, registered with
/// [`EditionSession::declare_edition`]. The set itself is computed from the registered
/// [`EditionInclusion`]s by [`EditionSession::encodings_in`].
#[derive(Clone, Copy, Debug)]
pub struct Edition {
/// The edition identifier. Also carries the freeze date: `core2026.07.0` freezes in
/// 2026-07.
pub id: EditionId,
/// The minimum Vortex version whose reader supports every encoding in this edition.
///
/// Recording this is the act of freezing: an edition with `None` is a **draft** — being
/// assembled, carrying no guarantee, free to change, never the default write target.
/// Validated against the members' [`EditionInclusion::required_vortex_release`] values:
/// no member may require a version newer than the edition declares.
pub min_vortex_version: Option<&'static str>,
}

impl Edition {
/// A draft is an edition whose `min_vortex_version` has not been recorded yet.
pub fn is_draft(&self) -> bool {
self.min_vortex_version.is_none()
}
}

/// Declares that an encoding is a member of an edition — and of every later edition of the
/// same family. Registered with [`EditionSession::declare_inclusion`].
#[derive(Clone, Copy, Debug)]
pub struct EditionInclusion {
/// The interned encoding id, e.g. `vortex.alp`. Globally unique across everything an
/// edition can cover: when layout encodings join editions, their ids must be distinct
/// from array encoding ids.
pub encoding_id: Id,
/// The first edition this encoding is a member of.
pub since: EditionId,
/// The earliest Vortex release able to read and execute this encoding, recorded from
/// evidence (e.g. compat-fixture history). `None` until recorded.
pub required_vortex_release: Option<&'static str>,
}

/// A source of an encoding id for edition declarations.
///
/// Implemented for raw id strings (`"vortex.alp"`) and interned [`Id`]s here; encoding
/// vtables implement it where they are defined, so a declaration can name the vtable
/// (`&Primitive`) instead of spelling its id.
pub trait AsEncodingId: Debug + Send + Sync {
/// The interned encoding id.
fn encoding_id(&self) -> Id;
}

impl AsEncodingId for str {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we use this impl anywhere?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that is used for each definition since they are defined by strs

#[expect(
clippy::disallowed_methods,
reason = "interning a dynamic encoding id at declaration time"
)]
fn encoding_id(&self) -> Id {
Id::new(self)
}
}

impl AsEncodingId for Id {
fn encoding_id(&self) -> Id {
*self
}
}

// `str` is unsized and cannot be a trait object, so declaration blocks (slices of
// `&dyn AsEncodingId`) name encodings as `&"vortex.alp"` through this impl.
impl AsEncodingId for &'static str {
fn encoding_id(&self) -> Id {
(**self).encoding_id()
}
}

/// Declares an edition together with the encodings that join the family at it, in one
/// block. Registered with [`EditionSession::declare`], which derives each encoding's
/// membership (`since` = the declared edition) from the block structure.
#[derive(Clone, Copy, Debug)]
pub struct EditionDeclaration {
/// The edition being declared.
pub edition: Edition,
/// The encodings that join the family at this edition, named by id string or by
/// vtable. Members of earlier editions are inherited and never restated.
pub added: &'static [&'static dyn AsEncodingId],
}

impl EditionInclusion {
/// Declare that an encoding is a member of `since` and every later edition of the same
/// family. The encoding can be named by id string or by vtable.
pub fn new<E: AsEncodingId + ?Sized>(encoding: &E, since: EditionId) -> Self {
Self {
encoding_id: encoding.encoding_id(),
since,
required_vortex_release: None,
}
}

/// Validate the declaration's form: a lowercase `namespace.name` encoding id and, if
/// recorded, a well-formed `major.minor.patch` release. Checked for every declared
/// inclusion by [`EditionSession::validate`].
pub fn validate(&self) -> Result<(), EditionError> {
let id = self.encoding_id.as_str();
let well_formed = !id.starts_with('.')
&& !id.ends_with('.')
&& id.contains('.')
&& id
.chars()
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || "._-".contains(c));
if !well_formed {
return Err(EditionError::new(format!(
"invalid encoding id {id:?}: expected lowercase `namespace.name`, e.g. \
`vortex.alp`"
)));
}
if let Some(release) = self.required_vortex_release
&& parse_release(release).is_none()
{
return Err(EditionError::new(format!(
"encoding {id} declares malformed required_vortex_release {release:?}"
)));
}
Ok(())
}
}

/// Parse a `major.minor.patch` release string into a comparable key.
pub(crate) fn parse_release(release: &str) -> Option<Vec<u64>> {
let parts: Vec<u64> = release
.split('.')
.map(|part| part.parse::<u64>().ok())
.collect::<Option<_>>()?;
(parts.len() == 3).then_some(parts)
}

/// Error raised when edition declarations are inconsistent.
#[derive(Debug)]
pub struct EditionError(String);

impl EditionError {
/// Create an error with the given message.
pub fn new(msg: impl Into<String>) -> Self {
Self(msg.into())
}
}

impl Display for EditionError {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
f.write_str(&self.0)
}
}

impl Error for EditionError {}
Loading
Loading