Skip to content

Commit c4cbb90

Browse files
claudeclaude
committed
feat(xtask): auto-generate the edition registry docs from the Rust declarations
Generate the edition registry pages from the edition declarations in `vortex/src/editions/`, in two stages: 1. A new `editions_manifest` example in the `vortex` crate validates the declarations and dumps them (with each edition's computed full encoding set) as a TOML manifest under `target/` — a build artifact, never committed. 2. A new `cargo xtask generate-editions-docs` subcommand runs the example, parses the manifest, and rewrites `docs/specs/editions/` wholesale: `index.md` with an editions table and a per-encoding index, plus one page per edition with its status, minimum Vortex reader, the encodings it adds, and its full encoding set. The static pages link into the registry: `docs/specs/editions.md` from its "Edition registry" section (previously a "Coming soon.." placeholder) and the unknown-encoding troubleshooting flow, and the docs landing page from a short static "Editions" section. The generated-files CI job now also runs `generate-editions-docs` before its "no files changed after regenerating" check, so a declaration change without a regen fails CI, including added or removed registry pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XKRaAHmFNvb7JoKTJVW8Pw Signed-off-by: Claude <noreply@anthropic.com>
1 parent 49c4753 commit c4cbb90

19 files changed

Lines changed: 673 additions & 2 deletions

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -644,6 +644,9 @@ jobs:
644644
run: |
645645
cargo run --profile ci -p xtask -- generate-fbs
646646
cargo run --profile ci -p xtask -- generate-proto
647+
- name: "regenerate the edition registry docs"
648+
run: |
649+
cargo run --profile ci -p xtask -- generate-editions-docs --profile ci
647650
- name: "regenerate FFI header file"
648651
run: |
649652
cargo +$NIGHTLY_TOOLCHAIN build --profile ci -p vortex-ffi

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -264,6 +264,7 @@ thiserror = "2.0.3"
264264
tokio = { version = "1.52" }
265265
tokio-stream = "0.1.17"
266266
tokio-util = "0.7.17"
267+
toml = "0.9.12"
267268
vortex-array-macros = { version = "0.1.0", path = "./vortex-array-macros" }
268269
# Pull these into non-public crates to support DF 58
269270
tpchgen = { version = "2.0.2", git = "https://github.com/clflushopt/tpchgen-rs.git", rev = "438e9c2dbc25b2fff82c0efc08b3f13b5707874f" }

‎docs/index.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,12 @@ internals. Build and benchmark locally.
9191
- **Language bindings**: First-class support for Python (PyO3), Java (JNI + Spark/Trino connectors),
9292
and C/C++ (FFI).
9393

94+
## Editions
95+
96+
Vortex writers target **[editions](specs/editions.md)** — named, frozen sets of encodings that
97+
carry a forever read-compatibility guarantee. See every edition in the
98+
[edition registry](specs/editions/index.md).
99+
94100
```{toctree}
95101
---
96102
hidden:

‎docs/specs/editions.md‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ editions is the earliest version of vortex required to read that file.
1919
## Resolving an unknown-encoding error
2020

2121
If a read failed with an unknown encoding ID and pointed you here, the reader met an encoding
22-
it does not support. Find the encoding ID in the [registry](#edition-registry) below:
22+
it does not support. Find the encoding ID in the [registry](editions/index.md):
2323

2424
1. **The ID is listed under an edition.** The file is newer than your Vortex build. Upgrade to
2525
at least that edition's required Vortex release and the file will read.
@@ -59,4 +59,14 @@ indefinitely, so deprecation never invalidates existing files.
5959

6060
## Edition registry
6161

62-
Coming soon..
62+
The full registry — every edition, its encoding set, and each encoding's membership — is
63+
generated from the declarations in `vortex/src/editions/` and published as its own pages,
64+
one per edition: see the [edition registry](editions/index.md).
65+
66+
```{toctree}
67+
---
68+
hidden:
69+
---
70+
71+
editions/index
72+
```

‎docs/specs/editions/core2025.05.0.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
<!-- Generated by `cargo xtask generate-editions-docs` from the declarations in `vortex/src/editions/`; do not edit. -->
2+
3+
# core2025.05.0
4+
5+
An edition of the `core` family. Frozen — readable by Vortex 0.36.0 and newer.
6+
7+
Adds 23 encodings: `fastlanes.bitpacked`, `fastlanes.for`, `vortex.alp`, `vortex.alprd`, `vortex.bool`, `vortex.bytebool`, `vortex.chunked`, `vortex.constant`, `vortex.datetimeparts`, `vortex.decimal`, `vortex.decimal_byte_parts`, `vortex.dict`, `vortex.ext`, `vortex.fsst`, `vortex.list`, `vortex.null`, `vortex.primitive`, `vortex.runend`, `vortex.sparse`, `vortex.struct`, `vortex.varbin`, `vortex.varbinview`, `vortex.zigzag`
8+
9+
## Encodings (23)
10+
11+
| Encoding | Since edition |
12+
| --- | --- |
13+
| `fastlanes.bitpacked` | [`core2025.05.0`](core2025.05.0.md) |
14+
| `fastlanes.for` | [`core2025.05.0`](core2025.05.0.md) |
15+
| `vortex.alp` | [`core2025.05.0`](core2025.05.0.md) |
16+
| `vortex.alprd` | [`core2025.05.0`](core2025.05.0.md) |
17+
| `vortex.bool` | [`core2025.05.0`](core2025.05.0.md) |
18+
| `vortex.bytebool` | [`core2025.05.0`](core2025.05.0.md) |
19+
| `vortex.chunked` | [`core2025.05.0`](core2025.05.0.md) |
20+
| `vortex.constant` | [`core2025.05.0`](core2025.05.0.md) |
21+
| `vortex.datetimeparts` | [`core2025.05.0`](core2025.05.0.md) |
22+
| `vortex.decimal` | [`core2025.05.0`](core2025.05.0.md) |
23+
| `vortex.decimal_byte_parts` | [`core2025.05.0`](core2025.05.0.md) |
24+
| `vortex.dict` | [`core2025.05.0`](core2025.05.0.md) |
25+
| `vortex.ext` | [`core2025.05.0`](core2025.05.0.md) |
26+
| `vortex.fsst` | [`core2025.05.0`](core2025.05.0.md) |
27+
| `vortex.list` | [`core2025.05.0`](core2025.05.0.md) |
28+
| `vortex.null` | [`core2025.05.0`](core2025.05.0.md) |
29+
| `vortex.primitive` | [`core2025.05.0`](core2025.05.0.md) |
30+
| `vortex.runend` | [`core2025.05.0`](core2025.05.0.md) |
31+
| `vortex.sparse` | [`core2025.05.0`](core2025.05.0.md) |
32+
| `vortex.struct` | [`core2025.05.0`](core2025.05.0.md) |
33+
| `vortex.varbin` | [`core2025.05.0`](core2025.05.0.md) |
34+
| `vortex.varbinview` | [`core2025.05.0`](core2025.05.0.md) |
35+
| `vortex.zigzag` | [`core2025.05.0`](core2025.05.0.md) |

‎docs/specs/editions/core2025.06.0.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
<!-- Generated by `cargo xtask generate-editions-docs` from the declarations in `vortex/src/editions/`; do not edit. -->
2+
3+
# core2025.06.0
4+
5+
An edition of the `core` family. Frozen — readable by Vortex 0.40.0 and newer.
6+
7+
Adds 3 encodings: `vortex.pco`, `vortex.sequence`, `vortex.zstd`
8+
9+
## Encodings (26)
10+
11+
| Encoding | Since edition |
12+
| --- | --- |
13+
| `fastlanes.bitpacked` | [`core2025.05.0`](core2025.05.0.md) |
14+
| `fastlanes.for` | [`core2025.05.0`](core2025.05.0.md) |
15+
| `vortex.alp` | [`core2025.05.0`](core2025.05.0.md) |
16+
| `vortex.alprd` | [`core2025.05.0`](core2025.05.0.md) |
17+
| `vortex.bool` | [`core2025.05.0`](core2025.05.0.md) |
18+
| `vortex.bytebool` | [`core2025.05.0`](core2025.05.0.md) |
19+
| `vortex.chunked` | [`core2025.05.0`](core2025.05.0.md) |
20+
| `vortex.constant` | [`core2025.05.0`](core2025.05.0.md) |
21+
| `vortex.datetimeparts` | [`core2025.05.0`](core2025.05.0.md) |
22+
| `vortex.decimal` | [`core2025.05.0`](core2025.05.0.md) |
23+
| `vortex.decimal_byte_parts` | [`core2025.05.0`](core2025.05.0.md) |
24+
| `vortex.dict` | [`core2025.05.0`](core2025.05.0.md) |
25+
| `vortex.ext` | [`core2025.05.0`](core2025.05.0.md) |
26+
| `vortex.fsst` | [`core2025.05.0`](core2025.05.0.md) |
27+
| `vortex.list` | [`core2025.05.0`](core2025.05.0.md) |
28+
| `vortex.null` | [`core2025.05.0`](core2025.05.0.md) |
29+
| `vortex.pco` | [`core2025.06.0`](core2025.06.0.md) |
30+
| `vortex.primitive` | [`core2025.05.0`](core2025.05.0.md) |
31+
| `vortex.runend` | [`core2025.05.0`](core2025.05.0.md) |
32+
| `vortex.sequence` | [`core2025.06.0`](core2025.06.0.md) |
33+
| `vortex.sparse` | [`core2025.05.0`](core2025.05.0.md) |
34+
| `vortex.struct` | [`core2025.05.0`](core2025.05.0.md) |
35+
| `vortex.varbin` | [`core2025.05.0`](core2025.05.0.md) |
36+
| `vortex.varbinview` | [`core2025.05.0`](core2025.05.0.md) |
37+
| `vortex.zigzag` | [`core2025.05.0`](core2025.05.0.md) |
38+
| `vortex.zstd` | [`core2025.06.0`](core2025.06.0.md) |

‎docs/specs/editions/core2025.10.0.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
<!-- Generated by `cargo xtask generate-editions-docs` from the declarations in `vortex/src/editions/`; do not edit. -->
2+
3+
# core2025.10.0
4+
5+
An edition of the `core` family. Frozen — readable by Vortex 0.54.0 and newer.
6+
7+
Adds 4 encodings: `fastlanes.rle`, `vortex.fixed_size_list`, `vortex.listview`, `vortex.masked`
8+
9+
## Encodings (30)
10+
11+
| Encoding | Since edition |
12+
| --- | --- |
13+
| `fastlanes.bitpacked` | [`core2025.05.0`](core2025.05.0.md) |
14+
| `fastlanes.for` | [`core2025.05.0`](core2025.05.0.md) |
15+
| `fastlanes.rle` | [`core2025.10.0`](core2025.10.0.md) |
16+
| `vortex.alp` | [`core2025.05.0`](core2025.05.0.md) |
17+
| `vortex.alprd` | [`core2025.05.0`](core2025.05.0.md) |
18+
| `vortex.bool` | [`core2025.05.0`](core2025.05.0.md) |
19+
| `vortex.bytebool` | [`core2025.05.0`](core2025.05.0.md) |
20+
| `vortex.chunked` | [`core2025.05.0`](core2025.05.0.md) |
21+
| `vortex.constant` | [`core2025.05.0`](core2025.05.0.md) |
22+
| `vortex.datetimeparts` | [`core2025.05.0`](core2025.05.0.md) |
23+
| `vortex.decimal` | [`core2025.05.0`](core2025.05.0.md) |
24+
| `vortex.decimal_byte_parts` | [`core2025.05.0`](core2025.05.0.md) |
25+
| `vortex.dict` | [`core2025.05.0`](core2025.05.0.md) |
26+
| `vortex.ext` | [`core2025.05.0`](core2025.05.0.md) |
27+
| `vortex.fixed_size_list` | [`core2025.10.0`](core2025.10.0.md) |
28+
| `vortex.fsst` | [`core2025.05.0`](core2025.05.0.md) |
29+
| `vortex.list` | [`core2025.05.0`](core2025.05.0.md) |
30+
| `vortex.listview` | [`core2025.10.0`](core2025.10.0.md) |
31+
| `vortex.masked` | [`core2025.10.0`](core2025.10.0.md) |
32+
| `vortex.null` | [`core2025.05.0`](core2025.05.0.md) |
33+
| `vortex.pco` | [`core2025.06.0`](core2025.06.0.md) |
34+
| `vortex.primitive` | [`core2025.05.0`](core2025.05.0.md) |
35+
| `vortex.runend` | [`core2025.05.0`](core2025.05.0.md) |
36+
| `vortex.sequence` | [`core2025.06.0`](core2025.06.0.md) |
37+
| `vortex.sparse` | [`core2025.05.0`](core2025.05.0.md) |
38+
| `vortex.struct` | [`core2025.05.0`](core2025.05.0.md) |
39+
| `vortex.varbin` | [`core2025.05.0`](core2025.05.0.md) |
40+
| `vortex.varbinview` | [`core2025.05.0`](core2025.05.0.md) |
41+
| `vortex.zigzag` | [`core2025.05.0`](core2025.05.0.md) |
42+
| `vortex.zstd` | [`core2025.06.0`](core2025.06.0.md) |

‎docs/specs/editions/core2026.07.0.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
<!-- Generated by `cargo xtask generate-editions-docs` from the declarations in `vortex/src/editions/`; do not edit. -->
2+
3+
# core2026.07.0
4+
5+
An edition of the `core` family. Frozen — readable by Vortex 0.65.0 and newer.
6+
7+
Adds 1 encoding: `vortex.variant`
8+
9+
## Encodings (31)
10+
11+
| Encoding | Since edition |
12+
| --- | --- |
13+
| `fastlanes.bitpacked` | [`core2025.05.0`](core2025.05.0.md) |
14+
| `fastlanes.for` | [`core2025.05.0`](core2025.05.0.md) |
15+
| `fastlanes.rle` | [`core2025.10.0`](core2025.10.0.md) |
16+
| `vortex.alp` | [`core2025.05.0`](core2025.05.0.md) |
17+
| `vortex.alprd` | [`core2025.05.0`](core2025.05.0.md) |
18+
| `vortex.bool` | [`core2025.05.0`](core2025.05.0.md) |
19+
| `vortex.bytebool` | [`core2025.05.0`](core2025.05.0.md) |
20+
| `vortex.chunked` | [`core2025.05.0`](core2025.05.0.md) |
21+
| `vortex.constant` | [`core2025.05.0`](core2025.05.0.md) |
22+
| `vortex.datetimeparts` | [`core2025.05.0`](core2025.05.0.md) |
23+
| `vortex.decimal` | [`core2025.05.0`](core2025.05.0.md) |
24+
| `vortex.decimal_byte_parts` | [`core2025.05.0`](core2025.05.0.md) |
25+
| `vortex.dict` | [`core2025.05.0`](core2025.05.0.md) |
26+
| `vortex.ext` | [`core2025.05.0`](core2025.05.0.md) |
27+
| `vortex.fixed_size_list` | [`core2025.10.0`](core2025.10.0.md) |
28+
| `vortex.fsst` | [`core2025.05.0`](core2025.05.0.md) |
29+
| `vortex.list` | [`core2025.05.0`](core2025.05.0.md) |
30+
| `vortex.listview` | [`core2025.10.0`](core2025.10.0.md) |
31+
| `vortex.masked` | [`core2025.10.0`](core2025.10.0.md) |
32+
| `vortex.null` | [`core2025.05.0`](core2025.05.0.md) |
33+
| `vortex.pco` | [`core2025.06.0`](core2025.06.0.md) |
34+
| `vortex.primitive` | [`core2025.05.0`](core2025.05.0.md) |
35+
| `vortex.runend` | [`core2025.05.0`](core2025.05.0.md) |
36+
| `vortex.sequence` | [`core2025.06.0`](core2025.06.0.md) |
37+
| `vortex.sparse` | [`core2025.05.0`](core2025.05.0.md) |
38+
| `vortex.struct` | [`core2025.05.0`](core2025.05.0.md) |
39+
| `vortex.varbin` | [`core2025.05.0`](core2025.05.0.md) |
40+
| `vortex.varbinview` | [`core2025.05.0`](core2025.05.0.md) |
41+
| `vortex.variant` | [`core2026.07.0`](core2026.07.0.md) |
42+
| `vortex.zigzag` | [`core2025.05.0`](core2025.05.0.md) |
43+
| `vortex.zstd` | [`core2025.06.0`](core2025.06.0.md) |

‎docs/specs/editions/index.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
<!-- Generated by `cargo xtask generate-editions-docs` from the declarations in `vortex/src/editions/`; do not edit. -->
2+
3+
# Edition registry
4+
5+
Every Vortex [edition](../editions.md), with one page per edition.
6+
7+
## Editions
8+
9+
| Edition | Status | Minimum Vortex reader | Encodings |
10+
| --- | --- | --- | --- |
11+
| [`core2025.05.0`](core2025.05.0.md) | frozen | 0.36.0 | 23 |
12+
| [`core2025.06.0`](core2025.06.0.md) | frozen | 0.40.0 | 26 |
13+
| [`core2025.10.0`](core2025.10.0.md) | frozen | 0.54.0 | 30 |
14+
| [`core2026.07.0`](core2026.07.0.md) | frozen | 0.65.0 | 31 |
15+
| [`unstable2025.05.0`](unstable2025.05.0.md) | draft | — | 1 |
16+
| [`unstable2026.02.0`](unstable2026.02.0.md) | draft | — | 2 |
17+
| [`unstable2026.04.0`](unstable2026.04.0.md) | draft | — | 8 |
18+
| [`unstable2026.06.0`](unstable2026.06.0.md) | draft | — | 9 |
19+
20+
A draft edition is still being assembled: its encoding set may change and it carries no compatibility guarantee until it is frozen.
21+
22+
## Encoding index
23+
24+
Each encoding lists the edition it first joined; it is a member of every later edition of the same family. The minimum Vortex reader is the one declared by that edition.
25+
26+
| Encoding | Since edition | Minimum Vortex reader |
27+
| --- | --- | --- |
28+
| `fastlanes.bitpacked` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
29+
| `fastlanes.delta` | [`unstable2025.05.0`](unstable2025.05.0.md) | — |
30+
| `fastlanes.for` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
31+
| `fastlanes.rle` | [`core2025.10.0`](core2025.10.0.md) | 0.54.0 |
32+
| `vortex.alp` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
33+
| `vortex.alprd` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
34+
| `vortex.bool` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
35+
| `vortex.bytebool` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
36+
| `vortex.chunked` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
37+
| `vortex.constant` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
38+
| `vortex.datetimeparts` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
39+
| `vortex.decimal` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
40+
| `vortex.decimal_byte_parts` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
41+
| `vortex.dict` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
42+
| `vortex.ext` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
43+
| `vortex.fixed_size_list` | [`core2025.10.0`](core2025.10.0.md) | 0.54.0 |
44+
| `vortex.fsst` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
45+
| `vortex.list` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
46+
| `vortex.listview` | [`core2025.10.0`](core2025.10.0.md) | 0.54.0 |
47+
| `vortex.masked` | [`core2025.10.0`](core2025.10.0.md) | 0.54.0 |
48+
| `vortex.null` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
49+
| `vortex.onpair` | [`unstable2026.06.0`](unstable2026.06.0.md) | — |
50+
| `vortex.parquet.variant` | [`unstable2026.04.0`](unstable2026.04.0.md) | — |
51+
| `vortex.patched` | [`unstable2026.04.0`](unstable2026.04.0.md) | — |
52+
| `vortex.pco` | [`core2025.06.0`](core2025.06.0.md) | 0.40.0 |
53+
| `vortex.primitive` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
54+
| `vortex.runend` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
55+
| `vortex.sequence` | [`core2025.06.0`](core2025.06.0.md) | 0.40.0 |
56+
| `vortex.sparse` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
57+
| `vortex.struct` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
58+
| `vortex.tensor.cosine_similarity` | [`unstable2026.04.0`](unstable2026.04.0.md) | — |
59+
| `vortex.tensor.inner_product` | [`unstable2026.04.0`](unstable2026.04.0.md) | — |
60+
| `vortex.tensor.l2_denorm` | [`unstable2026.04.0`](unstable2026.04.0.md) | — |
61+
| `vortex.tensor.l2_norm` | [`unstable2026.04.0`](unstable2026.04.0.md) | — |
62+
| `vortex.varbin` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
63+
| `vortex.varbinview` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
64+
| `vortex.variant` | [`core2026.07.0`](core2026.07.0.md) | 0.65.0 |
65+
| `vortex.zigzag` | [`core2025.05.0`](core2025.05.0.md) | 0.36.0 |
66+
| `vortex.zstd` | [`core2025.06.0`](core2025.06.0.md) | 0.40.0 |
67+
| `vortex.zstd_buffers` | [`unstable2026.02.0`](unstable2026.02.0.md) | — |
68+
69+
```{toctree}
70+
---
71+
hidden:
72+
---
73+
74+
core2025.05.0
75+
core2025.06.0
76+
core2025.10.0
77+
core2026.07.0
78+
unstable2025.05.0
79+
unstable2026.02.0
80+
unstable2026.04.0
81+
unstable2026.06.0
82+
```

0 commit comments

Comments
 (0)