Skip to content

Commit e4e5683

Browse files
dfa1claude
andcommitted
docs: reference.md catches up — layout SPI, id types, ColumnName
Also fixes fossil package FQNs in reference.md section headers (vortex.io / vortex.scan predate the current layout), adds the open(path, readRegistry, layoutRegistry) overload row, and freshens two CLAUDE.md lines (EncodingId is a sealed interface's WellKnown constant; zoned reads accept the vortex.zoned alias). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 9e6a146 commit e4e5683

2 files changed

Lines changed: 38 additions & 8 deletions

File tree

CLAUDE.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ Benchmark classes follow this: `JavaVsJni{Read,Write,Filter}Benchmark`,
2323

2424
```
2525
core — everything lives under `io.github.dfa1.vortex.core.*`:
26-
core.model DType, PType, TimeUnit, EncodingId, LayoutId, ExtensionId, TimeDtype, TimestampDtype
26+
core.model DType, PType, TimeUnit, EncodingId, LayoutId, ColumnName, ExtensionId, TimeDtype, TimestampDtype
2727
core.io IoBounds, PTypeIO, VortexFormat
2828
core.error VortexException
2929
core.compute FastLanes, PrimitiveArrays
@@ -119,15 +119,15 @@ DType (FlatBuffer), and Layout (FlatBuffer) blobs elsewhere in the file.
119119

120120
Layout tree: `Struct → Zoned(Stats) → Chunked → [Flat, Flat, ...]`
121121
- **Flat** single encoded segment · **Chunked** sequence of Flats · **Struct** one child/column
122-
- **Zoned** (`vortex.stats`) wraps a child with per-chunk min/max for zone-map pruning
122+
- **Zoned** (`vortex.stats`; reads also accept the newer Rust alias `vortex.zoned`) wraps a child with per-chunk min/max for zone-map pruning
123123

124124
Encoding IDs are strings (`"vortex.primitive"`, `"fastlanes.bitpacked"`). `ReadRegistry` maps IDs →
125125
`EncodingDecoder` via `ServiceLoader`; immutable after construction — register custom decoders on
126126
the builder: `ReadRegistry.builder().registerServiceLoaded().register(myDecoder).build()`.
127127

128128
### Adding an encoding
129129

130-
Add `EncodingId` enum constant `VORTEX_FOO("vortex.foo")`, then per side:
130+
Add an `EncodingId.WellKnown` constant `VORTEX_FOO("vortex.foo")` (re-exported on the interface), then per side:
131131
- **Decode:** `FooEncodingDecoder implements EncodingDecoder` in `reader.decode` + FQN in
132132
`reader/.../META-INF/services/io.github.dfa1.vortex.reader.decode.EncodingDecoder`
133133
- **Encode:** `FooEncodingEncoder implements EncodingEncoder` in `writer.encode` + FQN in

docs/reference.md

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,17 @@ shortcut returning a nullable copy), `withNullable(boolean)`, `DType.Struct.fiel
5555

5656
---
5757

58+
### Identity types (`io.github.dfa1.vortex.core.model`)
59+
60+
| Type | Shape | Notes |
61+
|------|-------|-------|
62+
| `EncodingId` | `sealed interface``WellKnown` enum + `Custom` record | Array-encoding identity; total `parse(String)` over non-blank ids; constants re-exported (`EncodingId.VORTEX_PRIMITIVE`, …) |
63+
| `LayoutId` | `sealed interface``WellKnown` enum + `Custom` record | Layout identity (separate namespace from encodings; `vortex.flat` is layout-only); both zoned aliases `vortex.zoned`/`vortex.stats` |
64+
| `ColumnName` | `record ColumnName(String value)` | Validated column name: non-blank, no control characters. `ColumnName.violation(String)` is the policy chokepoint shared by builder, writer, and file parser |
65+
5866
## Reader API
5967

60-
### `VortexReader` (`io.github.dfa1.vortex.io.VortexReader`)
68+
### `VortexReader` (`io.github.dfa1.vortex.reader.VortexReader`)
6169

6270
Memory-mapped handle to a Vortex file. Implements `AutoCloseable`. Closing releases the mmap region;
6371
all `Array` buffers obtained during scans become invalid.
@@ -66,6 +74,7 @@ all `Array` buffers obtained during scans become invalid.
6674
|---------------------------------------|---------------------------|-----------------------------------------------|
6775
| `static open(Path)` | `VortexReader` | Uses `ReadRegistry.loadAll()` |
6876
| `static open(Path, ReadRegistry)` | `VortexReader` | Custom registry (e.g. `allowUnknown()`) |
77+
| `static open(Path, ReadRegistry, LayoutRegistry)` | `VortexReader` | Custom layout decoders too |
6978
| `dtype()` | `DType` | Schema (typically `DType.Struct`) |
7079
| `layout()` | `Layout` | Layout tree (Struct → Zoned → Chunked → Flat) |
7180
| `footer()` | `Footer` | Segment specs, encoding specs |
@@ -127,7 +136,7 @@ Record: `(int chunkSize, boolean enableZoneMaps, double compressionRatioThreshol
127136

128137
## Scan API
129138

130-
### `ScanOptions` (`io.github.dfa1.vortex.scan.ScanOptions`)
139+
### `ScanOptions` (`io.github.dfa1.vortex.reader.ScanOptions`)
131140

132141
Record: `(List<String> columns, RowFilter rowFilter, long limit)`. Empty `columns` = read all. `NO_LIMIT` =
133142
`Long.MAX_VALUE`.
@@ -142,7 +151,7 @@ Record: `(List<String> columns, RowFilter rowFilter, long limit)`. Empty `column
142151
| `.withLimit(long n)` | Cap rows |
143152
| `.hasProjection()` / `.hasFilter()` / `.hasLimit()` | Predicates |
144153

145-
### `RowFilter` (`io.github.dfa1.vortex.scan.RowFilter`)
154+
### `RowFilter` (`io.github.dfa1.vortex.reader.RowFilter`)
146155

147156
Sealed predicate used for zone-map pruning (per-chunk min/max). Chunks that cannot match are skipped entirely.
148157

@@ -156,7 +165,7 @@ Sealed predicate used for zone-map pruning (per-chunk min/max). Chunks that cann
156165
| `RowFilter.Neq(column, value)` | `RowFilter.neq(col, val)` ||
157166
| `RowFilter.And(filters)` | `RowFilter.and(f1, f2, …)` | `f1.and(f2)` |
158167

159-
### `ScanIterator` (`io.github.dfa1.vortex.scan.ScanIterator`)
168+
### `ScanIterator` (`io.github.dfa1.vortex.reader.ScanIterator`)
160169

161170
Implements `Iterator<Chunk>` and `AutoCloseable`. Drives one scan.
162171

@@ -167,7 +176,7 @@ Implements `Iterator<Chunk>` and `AutoCloseable`. Drives one scan.
167176
| `forEachRemaining(Consumer)` | Overridden to wrap each `next()` in try-with-resources so chunks auto-close. |
168177
| `close()` | Releases iterator state and closes any chunk still open. |
169178

170-
### `Chunk` (`io.github.dfa1.vortex.scan.Chunk`)
179+
### `Chunk` (`io.github.dfa1.vortex.reader.Chunk`)
171180

172181
Implements `AutoCloseable`. Each chunk owns a confined `Arena` holding the decoded
173182
columnar buffers; closing the chunk releases the arena. After `close()`, touching
@@ -214,6 +223,27 @@ implementations are singletons invoked directly by their `ExtensionId`.
214223

215224
---
216225

226+
## Layout registry
227+
228+
### `LayoutRegistry` (`io.github.dfa1.vortex.reader.layout`)
229+
230+
Maps `LayoutId``LayoutDecoder`, making layout decode pluggable the way `ReadRegistry` makes
231+
encodings pluggable. Programmatic registration only — no `ServiceLoader`. Unknown layouts fail
232+
loudly (`VortexException`); there is no allow-unknown mode for layouts (Rust default).
233+
234+
| Method | Notes |
235+
|-----------------------------|--------------------------------------------------------------------|
236+
| `static defaults()` | The four built-ins: flat, chunked, zoned (both aliases), dict |
237+
| `static builder()` | Returns a fresh `Builder` |
238+
| `hasDecoder(LayoutId)` | Lookup |
239+
| `decode(ctx, layout, dtype)`| Dispatch by the layout's typed id |
240+
241+
`Builder`: `register(LayoutDecoder)` (throws on duplicate id), `registerDefaults()`, `build()`.
242+
A decoder claims its ids via `LayoutDecoder.layoutIds()` (a set — the zoned decoder claims both
243+
`vortex.zoned` and legacy `vortex.stats`). Custom decoders receive a `LayoutDecodeContext`
244+
(`decodeChild`, `decodeSegment` access, `arena`) and are reachable end-to-end via
245+
`VortexReader.open(path, readRegistry, layoutRegistry)`.
246+
217247
## Parquet / CSV import
218248

219249
### `ParquetImporter` (`io.github.dfa1.vortex.parquet.ParquetImporter`)

0 commit comments

Comments
 (0)