Skip to content

Commit 05c7c2f

Browse files
docs: introduce Vortex editions for users of the file format (#8855)
## Rationale for this change Vortex needs a named, user-visible compatibility contract for which encodings a writer may put in a file. This PR introduces **editions**: a named grouping of encodings that records exactly which encodings a Vortex file may contain and when each encoding joined the set. Writing under an edition guarantees every future Vortex release can read — and execute queries over — the file. The concept fills the forward/backward-compatibility seam already noted in `register_default_encodings` ("different Vortex 'Editions' that may support different sets of encodings") and gives unknown-encoding errors a stable docs page to link to. A follow-up PR adds the Rust implementation (`vortex-edition` crate) and the generated registry. --------- Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
1 parent 00cb75d commit 05c7c2f

3 files changed

Lines changed: 64 additions & 1 deletion

File tree

docs/specs/editions.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Editions
2+
3+
Vortex defines an evergrowing set of serializable array encodings, once written this can be read back by any future
4+
version of vortex.
5+
**Editions** are used to keep track of these encodings and talk about groups of encodings.
6+
7+
The edition `core2026.07.0` (coming soon) is the first such edition containing all encodings currently
8+
enabled by the writer.
9+
Editions are additive so an edition that comes after a previous one contains all the encodings from the previous one
10+
and more.
11+
The writer can be configured with a set of different editions (e.g. `core2026.07.0` and `unstable2026.05.0` all stable
12+
encoding released before July 2026 and all unstable encodings from May 2026).
13+
14+
Editions can be used to constrain your minimum required vortex reader, since latest version over vortex across all
15+
editions is the earliest version of vortex required to read that file.
16+
17+
## Resolving an unknown-encoding error
18+
19+
If a read failed with an unknown encoding ID and pointed you here, the reader met an encoding
20+
it does not support. Find the encoding ID in the [registry](#edition-registry) below:
21+
22+
1. **The ID is listed under an edition.** The file is newer than your Vortex build. Upgrade to
23+
at least that edition's required Vortex release and the file will read.
24+
2. **The ID is not listed anywhere.** The file was written outside the editions system, with a
25+
custom, third-party, or experimental encoding. Ask the producer of the file how to read it,
26+
or register the encoding with your session before reading. Tools that only inspect or
27+
relocate data (rather than query it) can opt in to `allow_unknown`, which decodes
28+
unrecognised encodings into inert placeholders.
29+
30+
## Writing with an edition
31+
32+
The default the writer targets a `core` edition lagging the latest vortex release by a few version giving delay before
33+
writing the latest vortex encodings to disk.
34+
Every file you write carries the read-forever guarantee. If a file would contain an encoding
35+
outside the targeted edition, the write fails immediately; edition violations never surface as
36+
someone else's read error later.
37+
38+
Two knobs exist when the default is not what you want:
39+
40+
- **Pin an older edition** when files must stay readable by deployments running older Vortex.
41+
- **Opt in to additional edition families.** Editions come in independently versioned,
42+
additive families — `core` today, with families for more specialised encoding groups (for
43+
example spatial encodings) possible later. A writer targets at most one edition per family
44+
and may emit any encoding in their union; each encoding belongs to exactly one family.
45+
46+
You can also opt out of editions entirely to write custom or experimental encodings. Doing so
47+
is an explicit choice that gives up the standardization guarantee — only readers that know your
48+
encodings can read those files.
49+
50+
## How editions change
51+
52+
A published edition is frozen — its encoding list never grows or shrinks. New encodings are
53+
staged in a **draft** edition and become guaranteed only when that draft is frozen as the next
54+
edition; each encoding's registry entry records the edition it joined in. In the future an
55+
encoding may be *deprecated*, meaning writers stop emitting it — but readers keep decoding it
56+
indefinitely, so deprecation never invalidates existing files.
57+
58+
## Edition registry
59+
60+
Coming soon..

docs/specs/file-format.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,9 @@ definition that allows efficiently querying the layout.
1717

1818
Other considerations for the Vortex file format include:
1919

20-
* Backwards compatibility, and (coming soon) forwards compatibility.
20+
* Backwards compatibility, and (coming soon) forwards compatibility. The set of encodings a
21+
writer may put in a file — and the resulting read-compatibility promise — is governed by
22+
[Editions](/specs/editions).
2123
* Fine-grained encryption.
2224
* Efficient access for both local disk and cloud storage.
2325
* Minimal overhead reading few columns or rows from wide or long arrays.

docs/specs/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ maxdepth: 2
88
---
99
1010
file-format
11+
editions
1112
ipc-format
1213
dtype-format
1314
scalar-format

0 commit comments

Comments
 (0)