Skip to content

CsvExporter renders a null struct row as {"field":null,...} instead of an empty cell #270

Description

@dfa1

Summary

A Struct-dtype column where the entire row is null (not just individual fields) renders as a JSON object with every field null (e.g. {"name":null,"count":null}) instead of an empty CSV cell, unlike every other nullable column type in CsvExporter.

Discovered while validating the fix for #268 (chunked List columns) against the Raincloud conformance corpus: oasst1 has a nullable struct column (labels) where some rows are entirely null. After #268's fix the vortex-java export no longer crashes, but its output diverges from the parquet oracle on those rows.

Root cause

StructEncodingDecoder (reader/src/main/java/io/github/dfa1/vortex/reader/decode/StructEncodingDecoder.java, ~line 55) represents struct-row nullability by wrapping each field in the same MaskedArray validity, not by wrapping the whole StructArray once:

return structValidity != null ? new MaskedArray(field, structValidity) : field;

So a null struct row decodes as a StructArray whose fields are each MaskedArray-wrapped-invalid — the StructArray itself is never masked.

CsvExporter.cellValue (csv/src/main/java/io/github/dfa1/vortex/csv/CsvExporter.java, ~line 165-196) only renders "" for a null value when it sees a MaskedArray wrapping the whole column value:

case MaskedArray ma -> ma.isValid(rowIdx) ? cellValue(ma.inner(), rowIdx) : "";
...
case StructArray sa -> jsonObject(sa, rowIdx);

A StructArray at the top of the switch always goes straight to jsonObject, which calls jsonValue per field — each field is individually masked invalid, so jsonValue's own MaskedArray arm renders "null" for each one, producing {"field1":null,"field2":null,...} instead of checking whether every field is invalid for this row (meaning the row itself is a null struct) and rendering "" instead, matching the parquet oracle's isNull(fieldIndex) → empty-cell behavior for a null top-level field.

Expected

cellValue's StructArray case (and/or jsonValue's, for a nested null struct inside a list/struct) should treat a struct row where every field is invalid as the row itself being null, and render an empty cell / JSON null respectively — matching every other nullable-column type in the same file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions