Skip to content

v0.12.2

Latest

Choose a tag to compare

@github-actions github-actions released this 12 Jul 07:41
Immutable release. Only release title and notes can be modified.

Registry cleanup, schema fidelity, and list-column support are the three themes of this
release. ReadRegistry and WriteRegistry are now builder-only (matching LayoutRegistry),
replacing ServiceLoader discovery with explicit registerDefaults() calls — registration is now
visible at the call site instead of hidden in META-INF/services files. The fbs-gen code
generator now strips trailing underscores from schema type names before prefixing with Fbs,
keeping generated Java identifiers clean (FbsStruct_FbsStruct) while staying wire-identical
to the upstream FlatBuffers schema. List/FixedSizeList columns gain full round-trip support:
VortexWriter now encodes vortex.list by default, CsvExporter renders both as JSON array cells
at any offset width, and ScanIterator can slice a shared list array whose covering flat chunk
spans several scan windows. Nullable Utf8/Binary columns now compress through the full cascade
instead of falling back to raw vortex.varbin, cutting file size 10–47× on the string-heavy
Raincloud corpus. Several correctness fixes land too: blank column names (produced by real
Raincloud corpus files) are now accepted on both paths; EncodingId/LayoutId reject control
characters; unsigned Parquet columns compare correctly; and six decoders receive tighter validation
against malformed input (ConstantEncodingDecoder, AlpRdEncodingDecoder, SparseEncodingDecoder,
DateTimePartsArrays, VarBinArray.DictMode, ScanIterator).

Added

  • CsvExporter now renders FixedSizeList and List columns as JSON array cells ([v0,v1,...]), matching the same element-rendering rules used for nested struct fields. Fixes export of Raincloud slugs with vector or list columns (e.g. glove-*, osm-germany-relations). (#257)
  • VortexWriter now encodes vortex.list columns by default (both flat and cascade paths). Previously only vortex.fixed_size_list was registered; DType.List columns threw at write time unless a custom registry was supplied. (#257)

Changed

  • fbs-gen strips trailing _ from schema names when generating Java class names (table Struct_FbsStruct, table Bool_FbsBool); the upstream FlatBuffers schema uses _ to avoid C++ reserved-name conflicts, which should not appear in Java. The runtime inline-struct base class is renamed FbsMemorySegment; FbsStruct is now the generated class for the Struct_ schema table. (84c912fe)
  • ReadRegistry and WriteRegistry are now builder-only (no ServiceLoader), matching LayoutRegistry. registerServiceLoaded() is replaced by registerDefaults(), which registers all built-in encoders/decoders explicitly; loadAll() is unchanged. ReadRegistry.decoderMap() is added. Custom encodings register via register(...) on the builder instead of a META-INF/services file. (#255)

Fixed

  • EncodingId.Custom and LayoutId.Custom now reject wire strings containing ISO control characters, mirroring ColumnName. A control-char id previously wrote a corrupt file that PostscriptParser crashes on at read time. (internal)
  • Parquet files with duplicate column names now compare correctly. The oracle header de-duplicates names with the Rust Vortex writer's algorithm (first occurrence unmodified, Nth repetition gets [N] suffix) and all row access uses column index rather than name. (#256)
  • Blank column names are now accepted on both the read and write paths. The Rust reference legitimately produces them (e.g. uci-electricityloaddiagrams20112014 has one at field index 0). ColumnName now enforces a symmetric policy: non-null and free of control characters (\n, \t, U+0000, …); blank names such as "" or " " pass through on both paths. (#255)
  • VarBinArray.DictMode reads dict-value offsets in a ptype-uniform loop: the I32 fast path eliminates the per-row switch(dictValOffPType) that blocked C2 vectorization (introduced by #215). (#243)
  • ConstantEncodingDecoder now returns NullArray for null-scalar constants (proto null_value tag); previously decoded as 0/false, silently corrupting nullable columns that the Rust writer encodes as an all-null constant. (#246)
  • ScanIterator.sliceArray now handles NullArray; previously threw on Rust-written files where a null-typed column used a coarser chunk grid than the other columns. (#247)
  • AlpRdEncodingDecoder now reads left_parts_ptype from metadata and rejects any value other than U16 with a VortexException; previously the field was silently ignored and U16 was hardcoded, risking a 2-byte stride over a 1-byte buffer for spec-conformant non-Rust writers. (#249)
  • SparseEncodingDecoder now asserts exactly two children (patch_indices, patch_values) and throws VortexException on any other count; previously a third child was silently accepted. (#250)
  • DateTimePartsArrays.readLong now zero-extends unsigned children (U8/U16/U32) instead of sign-extending them; previously a U16 seconds-within-day value ≥ 32 768 was widened as a negative short, producing a 65 536-second error in the reconstructed timestamp (#252, bi-euro2016 corpus).
  • RaincloudConformanceIntegrationTest now writes both the vortex-java and Parquet oracle CSVs to temp files and streams them line-by-line for comparison; previously it materialized both into heap String objects, causing OOM for large corpus files (≥157 MB) that were previously hidden by throwing before reaching the corrupted line. (#254)
  • RaincloudConformanceIntegrationTest oracle now formats INT32/INT64 Parquet columns with a UINT_32/UINT_64 logical-type annotation using Integer.toUnsignedString/Long.toUnsignedString, matching the unsigned representation that Vortex stores as U32/U64; previously the Parquet signed interpretation diverged for values above Integer.MAX_VALUE (e.g. total_boosters in the covid-world-vaccination-progress corpus). (#253)
  • RaincloudConformanceIntegrationTest now runs the Parquet oracle and vortex-java export in parallel via virtual threads connected by PipedWriter/PipedReader, comparing line-by-line without temp files; previously both CSV streams were written to disk and then compared, causing OOM for large corpus slugs (e.g. bi-bimbo with 74 M rows) and doubling wall time. (internal)
  • BoolEncodingDecoder now handles nullable vortex.bool columns: when the encoding node has one child, it is decoded as a validity bitmask and the values array is wrapped in a MaskedArray; previously the validity child was silently ignored and null rows were decoded as false, causing silent corruption in any corpus file with a nullable boolean column (e.g. bi-eixo st_extemporaneo_matricula). (internal)
  • Nullable Utf8/Binary columns are now compressed through the full cascade instead of stored as raw vortex.varbin. MaskedEncodingEncoder previously encoded its non-null values with a fixed first-match encoder (primitive/varbin), so a nullable low-cardinality string column never reached Dict or FSST — producing files 10–47× larger than the Rust reference on the string-heavy Raincloud corpus (e.g. uci-mushroom 22×, uci-online-retail-ii 25×). The masked values now run through the same CascadingCompressor as dense columns, dropping those ratios to ~2–3×. (#258)
  • ScanIterator can now slice a shared ListArray/FixedSizeListArray whose covering flat chunk spans several scan windows (misaligned per-column chunk grids). Previously any scan over such a column threw VortexException("scan: cannot slice shared array of type ...") — a real reading gap, not only an export-side one. (#265)
  • CsvExporter now reads vortex.list offsets of any integer width (I8/U8/I16/U16, matching VarBinArray's offsets), not only I32/I64. A narrower-offset list column (e.g. Raincloud's osm-germany-relations, whose offsets are I16) previously threw VortexException("unexpected list offsets type: ...") on export. (#263)