Registry cleanup, schema fidelity, and list-column support are the three themes of this
release. ReadRegistry and WriteRegistry are now builder-only (matching LayoutRegistry),
replacing ServiceLoader discovery with explicit registerDefaults() calls — registration is now
visible at the call site instead of hidden in META-INF/services files. The fbs-gen code
generator now strips trailing underscores from schema type names before prefixing with Fbs,
keeping generated Java identifiers clean (FbsStruct_ → FbsStruct) while staying wire-identical
to the upstream FlatBuffers schema. List/FixedSizeList columns gain full round-trip support:
VortexWriter now encodes vortex.list by default, CsvExporter renders both as JSON array cells
at any offset width, and ScanIterator can slice a shared list array whose covering flat chunk
spans several scan windows. Nullable Utf8/Binary columns now compress through the full cascade
instead of falling back to raw vortex.varbin, cutting file size 10–47× on the string-heavy
Raincloud corpus. Several correctness fixes land too: blank column names (produced by real
Raincloud corpus files) are now accepted on both paths; EncodingId/LayoutId reject control
characters; unsigned Parquet columns compare correctly; and six decoders receive tighter validation
against malformed input (ConstantEncodingDecoder, AlpRdEncodingDecoder, SparseEncodingDecoder,
DateTimePartsArrays, VarBinArray.DictMode, ScanIterator).
Added
CsvExporternow rendersFixedSizeListandListcolumns as JSON array cells ([v0,v1,...]), matching the same element-rendering rules used for nested struct fields. Fixes export of Raincloud slugs with vector or list columns (e.g.glove-*,osm-germany-relations). (#257)VortexWriternow encodesvortex.listcolumns by default (both flat and cascade paths). Previously onlyvortex.fixed_size_listwas registered;DType.Listcolumns threw at write time unless a custom registry was supplied. (#257)
Changed
fbs-genstrips trailing_from schema names when generating Java class names (table Struct_→FbsStruct,table Bool_→FbsBool); the upstream FlatBuffers schema uses_to avoid C++ reserved-name conflicts, which should not appear in Java. The runtime inline-struct base class is renamedFbsMemorySegment;FbsStructis now the generated class for theStruct_schema table. (84c912fe)ReadRegistryandWriteRegistryare now builder-only (noServiceLoader), matchingLayoutRegistry.registerServiceLoaded()is replaced byregisterDefaults(), which registers all built-in encoders/decoders explicitly;loadAll()is unchanged.ReadRegistry.decoderMap()is added. Custom encodings register viaregister(...)on the builder instead of aMETA-INF/servicesfile. (#255)
Fixed
EncodingId.CustomandLayoutId.Customnow reject wire strings containing ISO control characters, mirroringColumnName. A control-char id previously wrote a corrupt file thatPostscriptParsercrashes on at read time. (internal)- Parquet files with duplicate column names now compare correctly. The oracle header de-duplicates names with the Rust Vortex writer's algorithm (first occurrence unmodified, Nth repetition gets
[N]suffix) and all row access uses column index rather than name. (#256) - Blank column names are now accepted on both the read and write paths. The Rust reference legitimately produces them (e.g.
uci-electricityloaddiagrams20112014has one at field index 0).ColumnNamenow enforces a symmetric policy: non-null and free of control characters (\n,\t, U+0000, …); blank names such as""or" "pass through on both paths. (#255) VarBinArray.DictModereads dict-value offsets in a ptype-uniform loop: the I32 fast path eliminates the per-rowswitch(dictValOffPType)that blocked C2 vectorization (introduced by #215). (#243)ConstantEncodingDecodernow returnsNullArrayfor null-scalar constants (protonull_valuetag); previously decoded as0/false, silently corrupting nullable columns that the Rust writer encodes as an all-null constant. (#246)ScanIterator.sliceArraynow handlesNullArray; previously threw on Rust-written files where a null-typed column used a coarser chunk grid than the other columns. (#247)AlpRdEncodingDecodernow readsleft_parts_ptypefrom metadata and rejects any value other than U16 with aVortexException; previously the field was silently ignored and U16 was hardcoded, risking a 2-byte stride over a 1-byte buffer for spec-conformant non-Rust writers. (#249)SparseEncodingDecodernow asserts exactly two children (patch_indices,patch_values) and throwsVortexExceptionon any other count; previously a third child was silently accepted. (#250)DateTimePartsArrays.readLongnow zero-extends unsigned children (U8/U16/U32) instead of sign-extending them; previously aU16seconds-within-day value ≥ 32 768 was widened as a negative short, producing a 65 536-second error in the reconstructed timestamp (#252, bi-euro2016 corpus).RaincloudConformanceIntegrationTestnow writes both the vortex-java and Parquet oracle CSVs to temp files and streams them line-by-line for comparison; previously it materialized both into heapStringobjects, causing OOM for large corpus files (≥157 MB) that were previously hidden by throwing before reaching the corrupted line. (#254)RaincloudConformanceIntegrationTestoracle now formatsINT32/INT64Parquet columns with aUINT_32/UINT_64logical-type annotation usingInteger.toUnsignedString/Long.toUnsignedString, matching the unsigned representation that Vortex stores asU32/U64; previously the Parquet signed interpretation diverged for values aboveInteger.MAX_VALUE(e.g.total_boostersin the covid-world-vaccination-progress corpus). (#253)RaincloudConformanceIntegrationTestnow runs the Parquet oracle and vortex-java export in parallel via virtual threads connected byPipedWriter/PipedReader, comparing line-by-line without temp files; previously both CSV streams were written to disk and then compared, causing OOM for large corpus slugs (e.g. bi-bimbo with 74 M rows) and doubling wall time. (internal)BoolEncodingDecodernow handles nullablevortex.boolcolumns: when the encoding node has one child, it is decoded as a validity bitmask and the values array is wrapped in aMaskedArray; previously the validity child was silently ignored and null rows were decoded asfalse, causing silent corruption in any corpus file with a nullable boolean column (e.g. bi-eixost_extemporaneo_matricula). (internal)- Nullable Utf8/Binary columns are now compressed through the full cascade instead of stored as raw
vortex.varbin.MaskedEncodingEncoderpreviously encoded its non-null values with a fixed first-match encoder (primitive/varbin), so a nullable low-cardinality string column never reached Dict or FSST — producing files 10–47× larger than the Rust reference on the string-heavy Raincloud corpus (e.g.uci-mushroom22×,uci-online-retail-ii25×). The masked values now run through the sameCascadingCompressoras dense columns, dropping those ratios to ~2–3×. (#258) ScanIteratorcan now slice a sharedListArray/FixedSizeListArraywhose covering flat chunk spans several scan windows (misaligned per-column chunk grids). Previously any scan over such a column threwVortexException("scan: cannot slice shared array of type ...")— a real reading gap, not only an export-side one. (#265)CsvExporternow readsvortex.listoffsets of any integer width (I8/U8/I16/U16, matchingVarBinArray's offsets), not onlyI32/I64. A narrower-offset list column (e.g. Raincloud'sosm-germany-relations, whose offsets areI16) previously threwVortexException("unexpected list offsets type: ...")on export. (#263)