Summary
A Struct-dtype column where the entire row is null (not just individual fields) renders as a JSON object with every field null (e.g. {"name":null,"count":null}) instead of an empty CSV cell, unlike every other nullable column type in CsvExporter.
Discovered while validating the fix for #268 (chunked List columns) against the Raincloud conformance corpus: oasst1 has a nullable struct column (labels) where some rows are entirely null. After #268's fix the vortex-java export no longer crashes, but its output diverges from the parquet oracle on those rows.
Root cause
StructEncodingDecoder (reader/src/main/java/io/github/dfa1/vortex/reader/decode/StructEncodingDecoder.java, ~line 55) represents struct-row nullability by wrapping each field in the same MaskedArray validity, not by wrapping the whole StructArray once:
return structValidity != null ? new MaskedArray(field, structValidity) : field;
So a null struct row decodes as a StructArray whose fields are each MaskedArray-wrapped-invalid — the StructArray itself is never masked.
CsvExporter.cellValue (csv/src/main/java/io/github/dfa1/vortex/csv/CsvExporter.java, ~line 165-196) only renders "" for a null value when it sees a MaskedArray wrapping the whole column value:
case MaskedArray ma -> ma.isValid(rowIdx) ? cellValue(ma.inner(), rowIdx) : "";
...
case StructArray sa -> jsonObject(sa, rowIdx);
A StructArray at the top of the switch always goes straight to jsonObject, which calls jsonValue per field — each field is individually masked invalid, so jsonValue's own MaskedArray arm renders "null" for each one, producing {"field1":null,"field2":null,...} instead of checking whether every field is invalid for this row (meaning the row itself is a null struct) and rendering "" instead, matching the parquet oracle's isNull(fieldIndex) → empty-cell behavior for a null top-level field.
Expected
cellValue's StructArray case (and/or jsonValue's, for a nested null struct inside a list/struct) should treat a struct row where every field is invalid as the row itself being null, and render an empty cell / JSON null respectively — matching every other nullable-column type in the same file.
Summary
A
Struct-dtype column where the entire row is null (not just individual fields) renders as a JSON object with every fieldnull(e.g.{"name":null,"count":null}) instead of an empty CSV cell, unlike every other nullable column type inCsvExporter.Discovered while validating the fix for #268 (chunked List columns) against the Raincloud conformance corpus:
oasst1has a nullable struct column (labels) where some rows are entirely null. After #268's fix the vortex-java export no longer crashes, but its output diverges from the parquet oracle on those rows.Root cause
StructEncodingDecoder(reader/src/main/java/io/github/dfa1/vortex/reader/decode/StructEncodingDecoder.java, ~line 55) represents struct-row nullability by wrapping each field in the sameMaskedArrayvalidity, not by wrapping the wholeStructArrayonce:So a null struct row decodes as a
StructArraywhose fields are eachMaskedArray-wrapped-invalid — theStructArrayitself is never masked.CsvExporter.cellValue(csv/src/main/java/io/github/dfa1/vortex/csv/CsvExporter.java, ~line 165-196) only renders""for a null value when it sees aMaskedArraywrapping the whole column value:A
StructArrayat the top of the switch always goes straight tojsonObject, which callsjsonValueper field — each field is individually masked invalid, sojsonValue's ownMaskedArrayarm renders"null"for each one, producing{"field1":null,"field2":null,...}instead of checking whether every field is invalid for this row (meaning the row itself is a null struct) and rendering""instead, matching the parquet oracle'sisNull(fieldIndex)→ empty-cell behavior for a null top-level field.Expected
cellValue'sStructArraycase (and/orjsonValue's, for a nested null struct inside a list/struct) should treat a struct row where every field is invalid as the row itself being null, and render an empty cell / JSONnullrespectively — matching every other nullable-column type in the same file.