Skip to content

Commit f4209d0

Browse files
claudeclaude
committed
docs: introduce Vortex editions for users of the file format
An edition names the exact set of encodings a writer may put in a file, giving every file written under it a forever read-and-execute guarantee. This spec page explains the model for users of Vortex: editions constrain writers while readers are cumulative, family-namespaced editions (core2026.07.0) compose additively, writer presets choose within an edition without affecting portability, and unknown-encoding errors link back to the registry. Defines the first edition core2026.07.0 (the 32 encodings the default writer emits today) and the core2026.10.0 draft. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KdR42Svu74NcNzC7XJYwLr
1 parent c5e075a commit f4209d0

3 files changed

Lines changed: 132 additions & 1 deletion

File tree

‎docs/specs/editions.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Editions
2+
3+
An **edition** is a named grouping of encodings: it records exactly which encodings a Vortex
4+
file may contain, and when each encoding joined the set. The current edition,
5+
[`core2026.07.0`](#core2026070), groups the 32 encodings the default file writer emits today.
6+
7+
Editions make compatibility concrete. Because every edition is published and frozen, "written
8+
under `core2026.07.0`" says precisely which encodings a reader must support — and every Vortex
9+
release supports every edition published before it, so files written under an edition remain
10+
readable, and queryable, by all future versions of Vortex.
11+
12+
Editions constrain writing, not reading: a reader needs no configuration and always
13+
understands every published edition. The only time an edition surfaces on the read path is in
14+
the error described next.
15+
16+
## Resolving an unknown-encoding error
17+
18+
If a read failed with an unknown encoding ID and pointed you here, the reader met an encoding
19+
it does not support. Find the encoding ID in the [registry](#edition-registry) below:
20+
21+
1. **The ID is listed under an edition.** The file is newer than your Vortex build. Upgrade to
22+
at least that edition's required Vortex release and the file will read.
23+
2. **The ID is not listed anywhere.** The file was written outside the editions system, with a
24+
custom, third-party, or experimental encoding. Ask the producer of the file how to read it,
25+
or register the encoding with your session before reading. Tools that only inspect or
26+
relocate data (rather than query it) can opt in to `allow_unknown`, which decodes
27+
unrecognised encodings into inert placeholders.
28+
29+
## Writing with an edition
30+
31+
By default the writer targets the latest `core` edition — there is nothing to configure, and
32+
every file you write carries the read-forever guarantee. If a file would contain an encoding
33+
outside the targeted edition, the write fails immediately; edition violations never surface as
34+
someone else's read error later.
35+
36+
Two knobs exist when the default is not what you want:
37+
38+
- **Pin an older edition** when files must stay readable by deployments running older Vortex.
39+
- **Opt in to additional edition families.** Editions come in independently versioned,
40+
additive families — `core` today, with families for more specialised encoding groups (for
41+
example spatial encodings) possible later. A writer targets at most one edition per family
42+
and may emit any encoding in their union; each encoding belongs to exactly one family.
43+
44+
You can also opt out of editions entirely to write custom or experimental encodings. Doing so
45+
is an explicit choice that gives up the portability guarantee — only readers that know your
46+
encodings can read those files.
47+
48+
Compression presets (the default writer, the compact preset, CUDA-oriented presets) choose
49+
*within* the targeted edition, so switching presets never changes which readers can read a
50+
file.
51+
52+
## How editions change
53+
54+
A published edition is frozen — its encoding list never grows or shrinks. New encodings are
55+
staged in a **draft** edition and become guaranteed only when that draft is frozen as the next
56+
edition; each encoding's registry entry records the edition it joined in. In the future an
57+
encoding may be *deprecated*, meaning writers stop emitting it — but readers keep decoding it
58+
indefinitely, so deprecation never invalidates existing files.
59+
60+
Editions are also distinct from the file format version (the `u16` tag in the file's
61+
end-of-file marker): the format version describes the structure of the file container, the
62+
edition describes which encodings may appear inside it.
63+
64+
## Edition registry
65+
66+
| Edition | Status | Frozen on | Encodings |
67+
|---|---|---|---|
68+
| [core2026.07.0](#core2026070) | current | 2026-07 | 32 |
69+
| [core2026.10.0](#core2026100) | draft | — | 32 |
70+
71+
The registry is verified continuously: real `.vortex` files written by previous releases are
72+
re-read and checked against known-good values, and published editions are pinned by tests so
73+
no code change can silently alter a frozen set. The guarantee assumes a default-feature build
74+
of Vortex; disabling default cargo features (e.g. `zstd`) removes the corresponding decoders.
75+
76+
### core2026.07.0
77+
78+
| | |
79+
|---|---|
80+
| **Status** | current |
81+
| **Frozen on** | 2026-07 |
82+
| **Required Vortex release** | the first release that includes this edition |
83+
| **Encodings** | 32 |
84+
85+
| Encoding ID | Since |
86+
|---|---|
87+
| `fastlanes.bitpacked` | core2026.07.0 |
88+
| `fastlanes.delta` | core2026.07.0 |
89+
| `fastlanes.for` | core2026.07.0 |
90+
| `fastlanes.rle` | core2026.07.0 |
91+
| `vortex.alp` | core2026.07.0 |
92+
| `vortex.alprd` | core2026.07.0 |
93+
| `vortex.bool` | core2026.07.0 |
94+
| `vortex.bytebool` | core2026.07.0 |
95+
| `vortex.chunked` | core2026.07.0 |
96+
| `vortex.constant` | core2026.07.0 |
97+
| `vortex.datetimeparts` | core2026.07.0 |
98+
| `vortex.decimal` | core2026.07.0 |
99+
| `vortex.decimal_byte_parts` | core2026.07.0 |
100+
| `vortex.dict` | core2026.07.0 |
101+
| `vortex.ext` | core2026.07.0 |
102+
| `vortex.fixed_size_list` | core2026.07.0 |
103+
| `vortex.fsst` | core2026.07.0 |
104+
| `vortex.list` | core2026.07.0 |
105+
| `vortex.listview` | core2026.07.0 |
106+
| `vortex.masked` | core2026.07.0 |
107+
| `vortex.null` | core2026.07.0 |
108+
| `vortex.pco` | core2026.07.0 |
109+
| `vortex.primitive` | core2026.07.0 |
110+
| `vortex.runend` | core2026.07.0 |
111+
| `vortex.sequence` | core2026.07.0 |
112+
| `vortex.sparse` | core2026.07.0 |
113+
| `vortex.struct` | core2026.07.0 |
114+
| `vortex.varbin` | core2026.07.0 |
115+
| `vortex.varbinview` | core2026.07.0 |
116+
| `vortex.variant` | core2026.07.0 |
117+
| `vortex.zigzag` | core2026.07.0 |
118+
| `vortex.zstd` | core2026.07.0 |
119+
120+
Not included, because they are experimental and their serialized forms may still change:
121+
`vortex.onpair`, `vortex.zstd_buffers`, `vortex.patched`, `vortex.piecewise-sequence`, and
122+
everything behind the `unstable_encodings` feature flag. Writing them requires opting out of
123+
the edition.
124+
125+
### core2026.10.0
126+
127+
The draft staging area for the next `core` edition: no guarantee until frozen, contents may
128+
change freely. No changes relative to [core2026.07.0](#core2026070) yet.

‎docs/specs/file-format.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,9 @@ definition that allows efficiently querying the layout.
1717

1818
Other considerations for the Vortex file format include:
1919

20-
* Backwards compatibility, and (coming soon) forwards compatibility.
20+
* Backwards compatibility, and (coming soon) forwards compatibility. The set of encodings a
21+
writer may put in a file — and the resulting read-compatibility promise — is governed by
22+
[Editions](/specs/editions).
2123
* Fine-grained encryption.
2224
* Efficient access for both local disk and cloud storage.
2325
* Minimal overhead reading few columns or rows from wide or long arrays.

‎docs/specs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ maxdepth: 2
88
---
99
1010
file-format
11+
editions
1112
ipc-format
1213
dtype-format
1314
scalar-format

0 commit comments

Comments
 (0)