Skip to content

Commit 47a91fd

Browse files
dfa1claude
dfa1
and
claude
committed
docs(compatibility): add Decode shape table — zero-copy / lazy / materialized
For every encoding, document the current decode-output shape (zero-copy view / lazy record / materialized buffer) plus the target final state per ADRs 0010 and 0012. Clarifies which encodings are already lazy, which are still materialised but should be lazy, and which (decompression encodings) stay materialised by design. Surfaces the remaining ADR 0012 gap: vortex.dict and vortex.runend / vortex.rle / vortex.sparse still materialise and have a path to Lazy. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent d048674 commit 47a91fd

1 file changed

Lines changed: 57 additions & 0 deletions

File tree

‎docs/compatibility.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,63 @@ resolves only the standalone decoders in `reader`; no encoder class is loaded.
7575
| `vortex.variant` | `VariantEncodingDecoder` | `VariantEncodingEncoder` | ✅ | ❌ | Decode (incl. shredded child); encode not yet implemented (Rust 0.73+) |
7676
| `vortex.onpair` | _none_ | _none_ | ❌ | ❌ | Experimental in Rust 0.74.0; not yet ported |
7777

78+
### Decode shape
79+
80+
Per [ADR 0010](adr/0010-lazy-decode.md) and [ADR 0012](adr/0012-zero-copy-layout-decoding.md), each
81+
decoder falls into one of three shapes:
82+
83+
- **Zero-copy** — output is a view over the memory-mapped file (or a wrapper over child arrays).
84+
No arena allocation, no per-element copy.
85+
- **Lazy** — output is a `LazyXxxArray` / `ChunkedXxxArray` record that holds the encoded child
86+
plus the transform parameters. Per-row `getXxx(i)` applies the transform on demand. No
87+
output buffer is allocated unless a caller explicitly materialises via
88+
`ArraySegments.of(arr, arena)`.
89+
- **Materialized** — output is a buffer allocated from `ctx.arena()` populated during `decode()`.
90+
Required for decompression-style encodings (Bitpacked, Pco, Zstd, etc.) where reading element
91+
`i` would require decoding a window.
92+
93+
| Encoding ID | Now | Target | Notes |
94+
|-----------------------------|---------------|---------------|--------------------------------------------------------------------------|
95+
| `vortex.primitive` | Zero-copy | Zero-copy | mmap slice |
96+
| `vortex.bool` | Zero-copy | Zero-copy | mmap slice (bit-packed) |
97+
| `vortex.null` | n/a | n/a | no per-row data |
98+
| `vortex.bytebool` | Zero-copy | Zero-copy | mmap slice |
99+
| `vortex.zigzag` | Lazy | Lazy | `LazyZigZagXxxArray`, ADR 0010 |
100+
| `vortex.constant` | Zero-copy | Zero-copy | single-element broadcast wrapper |
101+
| `vortex.ext` | Zero-copy | Zero-copy | wraps storage |
102+
| `vortex.runend` | Materialized | Lazy | could expose run-locating accessor (similar to Dict) |
103+
| `vortex.varbin` | Zero-copy | Zero-copy | bytes + offsets slices |
104+
| `vortex.varbinview` | Materialized | TBD | currently converts to `varbin` shape |
105+
| `vortex.alp` | Lazy | Lazy | `LazyAlpDoubleArray`/`LazyAlpFloatArray`, ADR 0010 |
106+
| `vortex.alprd` | Materialized | TBD | reassembles left/right + patches |
107+
| `vortex.dict` | Materialized | Lazy | ADR 0012 Dict\*Array, **NOT YET IMPLEMENTED** |
108+
| `vortex.sparse` | Materialized | Lazy | patches view over default fill |
109+
| `vortex.sequence` | Zero-copy | Zero-copy | synthetic (no data) |
110+
| `vortex.struct` | Zero-copy | Zero-copy | `StructArray` wraps fields |
111+
| `vortex.chunked` | Lazy | Lazy | `ChunkedXxxArray`, ADR 0012 (PR #38) |
112+
| `vortex.fsst` | Materialized | Materialized | symbol-table decompression |
113+
| `vortex.list` | Materialized | TBD | offsets handling |
114+
| `vortex.listview` | Materialized | TBD | similar to list |
115+
| `vortex.fixed_size_list` | Materialized | TBD | inline elements |
116+
| `vortex.zstd` | Materialized | Materialized | block decompression |
117+
| `vortex.masked` | Zero-copy | Zero-copy | wraps inner + validity |
118+
| `vortex.decimal` | Materialized | TBD | converts to `BigDecimal` |
119+
| `vortex.decimal_byte_parts` | Materialized | TBD | reassembles byte parts |
120+
| `vortex.datetimeparts` | Materialized | TBD | reassembles parts |
121+
| `vortex.pco` | Materialized | Materialized | range-encoded decompression |
122+
| `fastlanes.bitpacked` | Materialized | Materialized | window unpacks bits |
123+
| `fastlanes.delta` | Materialized | Materialized | cumulative sum requires sequential decode |
124+
| `fastlanes.for` | Lazy | Lazy | `LazyForLongArray`/`LazyForIntArray`, ADR 0010 |
125+
| `fastlanes.rle` | Materialized | Lazy | run-locating accessor possible |
126+
| `vortex.patched` | Materialized | TBD | scatter patches |
127+
| `vortex.variant` | Materialized | TBD | shredded child reassembly |
128+
| `vortex.onpair` | n/a | n/a | not ported |
129+
130+
Decompression-style encodings (Bitpacked / Pco / Zstd / Fsst / Delta) stay Materialized by design
131+
— element-at-`i` requires decoding a window, so they must allocate output (ADR 0010 §"Decompression
132+
encodings stay eager"). Their output can itself be wrapped in a 1:1 lazy transform (e.g. ALP over
133+
Bitpacked produces `LazyAlp(MaterializedXxx)`).
134+
78135
### Unknown encodings
79136

80137
Files containing unrecognised encoding IDs throw `VortexException` by default. Opt in to

0 commit comments

Comments
 (0)