fix: render nested struct columns as JSON object cells in CSV export (#217)

A nested struct column (StructArray, decodable since #207) now renders
as one JSON object cell {"field":value,...} with fields in declared
order: numbers/booleans unquoted via the shared leaf rules, strings
JSON-escaped and quoted, nested structs recursed, and null fields as
JSON null (inside JSON the bare-CSV empty-field convention would be
ambiguous). Only the JSON layer escapes; fastcsv independently quotes
the cell. VortexWriter.arrayLength learns StructData so struct columns
can be written in tests.

Pinned by countries-of-the-world (Raincloud corpus, #205): its data
struct column blocked CSV export after the #207 scan fix.

Closes #217

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: dfa1

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- CSV export renders nested struct columns as JSON object cells (`{"field":value,...}`, strings JSON-escaped, null fields as JSON `null`, nested structs recursed) instead of throwing `unsupported array type: StructArray`. ([#217](https://github.com/dfa1/vortex-java/issues/217))
 - Scanning a struct whose columns include a nested struct no longer fails chunk planning. A nested `vortex.struct` layout column (e.g. Raincloud `countries-of-the-world`'s `data` column) is now treated as a single full-range chunk source and decoded through the layout registry's new struct decoder into a `StructArray`, matching the Rust reference (a nested struct spans its parent's row range, not an independent chunking). `schema`/`count`/`inspect` already worked; plain scans and `select` of sibling columns now work too. CSV export of the struct column itself remains unsupported (a separate rendering limitation). ([#207](https://github.com/dfa1/vortex-java/issues/207))
 - CSV export renders unsigned integer columns (U8–U64) with their unsigned values — high-half values previously printed as two's-complement negatives (uci-wine `magnesium` U8 132 exported as -124), silent corruption found by the Raincloud conformance suite. ([#208](https://github.com/dfa1/vortex-java/issues/208))
 - Unsigned integer columns are handled unsigned in the remaining consumers: the CLI `filter` predicates (`magnesium >= 130` now matches its high-half U8 rows instead of silently dropping them — the worst class of the bug: wrong query results), the TUI grid and inspector views, and the Calcite adapter (U8/U16/U32 map to the next wider signed SQL type so their full range fits; U64 stays `BIGINT` and fails loud rather than surfacing a negative for values ≥ 2^63, both when read and when summed). ([#216](https://github.com/dfa1/vortex-java/issues/216))

csv/src/main/java/io/github/dfa1/vortex/csv/CsvExporter.java

@@ -14,6 +14,7 @@
 import io.github.dfa1.vortex.reader.array.ShortArray;
 import io.github.dfa1.vortex.reader.array.MaskedArray;
 import io.github.dfa1.vortex.reader.array.NullArray;
+import io.github.dfa1.vortex.reader.array.StructArray;
 import io.github.dfa1.vortex.reader.array.VarBinArray;
 import io.github.dfa1.vortex.reader.VortexReader;
 import io.github.dfa1.vortex.reader.ScanIterator;
@@ -139,6 +140,22 @@ private static void writeChunk(Chunk chunk, List<String> colNames, int colCount,
         state[1] = nextNotify;
     }
 
+    /// Renders one cell for the row `rowIdx` of column array `arr`.
+    ///
+    /// Leaf columns follow the JDK canonical rendering per type (signed/unsigned integer decimal,
+    /// `Double.toString`, `Boolean.toString`, the raw utf8 string); a null row (a [MaskedArray]
+    /// invalid row or a [NullArray]) renders as an empty CSV field.
+    ///
+    /// A nested struct column ([StructArray], possibly wrapped in a [MaskedArray] when nullable)
+    /// renders as a single JSON object cell `{"field":value,...}` with fields in the struct dtype's
+    /// declared order. Field values reuse the same leaf rendering rules — numbers and booleans
+    /// unquoted, strings JSON-escaped and double-quoted, nested structs recursed — and a null field
+    /// (or null nested row) becomes a JSON `null`. Only the JSON layer is escaped here; the CSV
+    /// writer independently quotes any cell containing the delimiter or a quote character.
+    ///
+    /// @param arr    the column array to read from
+    /// @param rowIdx the zero-based row index within `arr`
+    /// @return the rendered cell text
     private static String cellValue(Array arr, long rowIdx) {
         return switch (arr) {
             // Long/IntArray have no dtype-aware getter, so gate the unsigned rendering on the
@@ -163,11 +180,79 @@ private static String cellValue(Array arr, long rowIdx) {
             // All-null columns (DType.Null) hold only a row count: every cell is an empty
             // field, same rule as a MaskedArray null row.
             case NullArray ignored -> "";
+            // Nested struct column: render the whole row as a JSON object cell.
+            case StructArray sa -> jsonObject(sa, rowIdx);
             default -> throw new VortexException(
                     "unsupported array type for CSV export: " + arr.getClass().getSimpleName());
         };
     }
 
+    /// Renders one struct row as a JSON object `{"field":value,...}` with fields in the struct
+    /// dtype's declared order.
+    private static String jsonObject(StructArray struct, long rowIdx) {
+        DType.Struct dtype = (DType.Struct) struct.dtype();
+        List<ColumnName> names = dtype.fieldNames();
+        StringBuilder sb = new StringBuilder();
+        sb.append('{');
+        for (int i = 0; i < names.size(); i++) {
+            if (i > 0) {
+                sb.append(',');
+            }
+            jsonString(sb, names.get(i).value());
+            sb.append(':');
+            sb.append(jsonValue(struct.field(i), rowIdx));
+        }
+        sb.append('}');
+        return sb.toString();
+    }
+
+    /// Renders one struct field value as JSON text: a nested object for structs, a quoted escaped
+    /// string for utf8, JSON `null` for a null row/field, and the bare leaf rendering (unquoted
+    /// number/boolean) for everything else. Unlike [#cellValue(Array, long)], a null here is the
+    /// JSON token `null` rather than an empty field, because inside a JSON object the empty-field
+    /// convention of bare CSV would be ambiguous.
+    private static String jsonValue(Array arr, long rowIdx) {
+        return switch (arr) {
+            case StructArray sa -> jsonObject(sa, rowIdx);
+            case MaskedArray ma -> ma.isValid(rowIdx) ? jsonValue(ma.inner(), rowIdx) : "null";
+            case NullArray ignored -> "null";
+            case VarBinArray va -> {
+                StringBuilder sb = new StringBuilder();
+                jsonString(sb, va.getString(rowIdx));
+                yield sb.toString();
+            }
+            // Numeric and boolean leaves render identically to a top-level cell, unquoted.
+            default -> cellValue(arr, rowIdx);
+        };
+    }
+
+    /// Appends `value` to `sb` as a JSON string literal: wrapped in double quotes with `"`,
+    /// backslash, the standard short escapes, and any other control character below `U+0020`
+    /// escaped as `\\uXXXX`.
+    private static void jsonString(StringBuilder sb, String value) {
+        sb.append('"');
+        for (int i = 0; i < value.length(); i++) {
+            char c = value.charAt(i);
+            switch (c) {
+                case '"' -> sb.append("\\\"");
+                case '\\' -> sb.append("\\\\");
+                case '\n' -> sb.append("\\n");
+                case '\r' -> sb.append("\\r");
+                case '\t' -> sb.append("\\t");
+                case '\b' -> sb.append("\\b");
+                case '\f' -> sb.append("\\f");
+                default -> {
+                    if (c < 0x20) {
+                        sb.append(String.format("\\u%04x", (int) c));
+                    } else {
+                        sb.append(c);
+                    }
+                }
+            }
+        }
+        sb.append('"');
+    }
+
     /// Whether the array's dtype is an unsigned integer, so high-half values must render as
     /// unsigned decimal rather than the two's-complement negative that the raw signed getter
     /// returns. Callers only pass leaf value arrays: the [MaskedArray] arm of `cellValue`

csv/src/test/java/io/github/dfa1/vortex/csv/CsvExporterTest.java

@@ -5,9 +5,14 @@
 import io.github.dfa1.vortex.core.model.PType;
 import io.github.dfa1.vortex.writer.VortexWriter;
 import io.github.dfa1.vortex.writer.WriteOptions;
+import io.github.dfa1.vortex.writer.encode.BoolEncodingEncoder;
+import io.github.dfa1.vortex.writer.encode.MaskedEncodingEncoder;
 import io.github.dfa1.vortex.writer.encode.NullEncodingEncoder;
 import io.github.dfa1.vortex.writer.encode.NullableData;
 import io.github.dfa1.vortex.writer.encode.PrimitiveEncodingEncoder;
+import io.github.dfa1.vortex.writer.encode.StructData;
+import io.github.dfa1.vortex.writer.encode.StructEncodingEncoder;
+import io.github.dfa1.vortex.writer.encode.VarBinEncodingEncoder;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.io.TempDir;
 
@@ -259,4 +264,160 @@ void rendersU64AboveLongMaxAsUnsigned(@TempDir Path tmp) throws Exception {
         List<String> result = Files.readAllLines(csv);
         assertThat(result).containsExactly("v", "1", "9223372036854775808", "18446744073709551615");
     }
+
+    @Test
+    void rendersNestedStructColumnAsJsonObject(@TempDir Path tmp) throws Exception {
+        // Given a struct column of mixed field types (i64, utf8, f64), the shape #217 threw on:
+        // StructArray had no cellValue arm. The sibling plain columns confirm structs coexist
+        // with scalar columns in the same file.
+        Path vortex = tmp.resolve("nested.vortex");
+        DType.Struct inner = new DType.Struct(
+                List.of(ColumnName.of("id"), ColumnName.of("name"), ColumnName.of("score")),
+                List.of(DType.I64, DType.UTF8, DType.F64),
+                false);
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("region"), ColumnName.of("data")),
+                List.of(DType.UTF8, inner),
+                false);
+        try (FileChannel ch = FileChannel.open(vortex, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             // Struct columns route through the recursive cascade (the flat first-match path has
+             // no struct encoder), so enable cascading here and in the other struct cases.
+             VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.cascading(3))) {
+            writer.writeChunk(Map.of(
+                    ColumnName.of("region"), new String[]{"north", "south"},
+                    ColumnName.of("data"), new StructData(List.of(
+                            new long[]{1L, 2L},
+                            new String[]{"Alice", "Bob"},
+                            new double[]{9.5, 3.25}))));
+        }
+        Path csv = tmp.resolve("out.csv");
+
+        // When
+        CsvExporter.exportCsv(vortex, csv);
+
+        // Then — the struct cell is a JSON object in declared field order; numbers/strings quoted
+        // per JSON, not CSV.
+        List<String> result = Files.readAllLines(csv);
+        assertThat(result).containsExactly(
+                "region,data",
+                "north,\"{\"\"id\"\":1,\"\"name\"\":\"\"Alice\"\",\"\"score\"\":9.5}\"",
+                "south,\"{\"\"id\"\":2,\"\"name\"\":\"\"Bob\"\",\"\"score\"\":3.25}\"");
+    }
+
+    @Test
+    void rendersNullStructFieldAsJsonNull(@TempDir Path tmp) throws Exception {
+        // Given a struct whose second field is nullable and null on the first row — the field must
+        // become the JSON token null (not an empty field, which is bare CSV's null convention and
+        // ambiguous inside a JSON object). Written on the flat path with an explicit struct encoder:
+        // StructEncodingEncoder wraps a NullableData field in the masked encoder, whereas the
+        // recursive cascade does not yet accept nullable struct fields. A `region` sibling keeps the
+        // struct column off the single-column path; `label` needs two struct fields to survive decode
+        // (a one-field struct is unwrapped to its bare field by StructEncodingDecoder).
+        Path vortex = tmp.resolve("nullfield.vortex");
+        DType.Struct inner = new DType.Struct(
+                List.of(ColumnName.of("id"), ColumnName.of("label")),
+                List.of(DType.I64, new DType.Primitive(PType.I64, true)),
+                false);
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("region"), ColumnName.of("data")),
+                List.of(DType.UTF8, inner),
+                false);
+        try (FileChannel ch = FileChannel.open(vortex, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.defaults(),
+                     List.of(new StructEncodingEncoder(), new PrimitiveEncodingEncoder(),
+                             new VarBinEncodingEncoder(), new MaskedEncodingEncoder(),
+                             new BoolEncodingEncoder()))) {
+            writer.writeChunk(Map.of(
+                    ColumnName.of("region"), new String[]{"north", "south"},
+                    ColumnName.of("data"), new StructData(List.of(
+                            new long[]{1L, 2L},
+                            new NullableData(new long[]{0L, 42L}, new boolean[]{false, true})))));
+        }
+        Path csv = tmp.resolve("out.csv");
+
+        // When
+        CsvExporter.exportCsv(vortex, csv);
+
+        // Then the null field is the JSON token null; the valid field is the bare number.
+        List<String> result = Files.readAllLines(csv);
+        assertThat(result).containsExactly(
+                "region,data",
+                "north,\"{\"\"id\"\":1,\"\"label\"\":null}\"",
+                "south,\"{\"\"id\"\":2,\"\"label\"\":42}\"");
+    }
+
+    @Test
+    void escapesQuotesAndCommasInsideStructStringField(@TempDir Path tmp) throws Exception {
+        // Given a struct string field value containing a JSON-significant quote and a comma: the
+        // quote must be JSON-escaped (\") inside the object, and the CSV layer independently doubles
+        // the quotes of the whole cell. The comma must NOT split the CSV cell. Two struct fields
+        // (note, rank) keep the struct from being unwrapped on decode; the `label` sibling keeps it
+        // off the single-column path.
+        Path vortex = tmp.resolve("escape.vortex");
+        DType.Struct inner = new DType.Struct(
+                List.of(ColumnName.of("note"), ColumnName.of("rank")),
+                List.of(DType.UTF8, DType.I64),
+                false);
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("label"), ColumnName.of("data")),
+                List.of(DType.UTF8, inner),
+                false);
+        try (FileChannel ch = FileChannel.open(vortex, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.cascading(3))) {
+            writer.writeChunk(Map.of(
+                    ColumnName.of("label"), new String[]{"x"},
+                    ColumnName.of("data"), new StructData(List.of(
+                            new String[]{"a\"b,c"},
+                            new long[]{7L}))));
+        }
+        Path csv = tmp.resolve("out.csv");
+
+        // When
+        CsvExporter.exportCsv(vortex, csv);
+
+        // Then — JSON escapes the inner quote to \"; the CSV layer then doubles every quote in the
+        // whole field. The comma stays inside the single quoted cell.
+        List<String> result = Files.readAllLines(csv);
+        assertThat(result).containsExactly(
+                "label,data",
+                "x,\"{\"\"note\"\":\"\"a\\\"\"b,c\"\",\"\"rank\"\":7}\"");
+    }
+
+    @Test
+    void rendersStructInStructAsNestedJsonObject(@TempDir Path tmp) throws Exception {
+        // Given a struct whose field is itself a struct: the inner struct must recurse into a nested
+        // JSON object, not throw. A `region` sibling keeps the struct column off the single-column
+        // path; every struct here has two fields so none is unwrapped on decode.
+        Path vortex = tmp.resolve("deep.vortex");
+        DType.Struct innermost = new DType.Struct(
+                List.of(ColumnName.of("lat"), ColumnName.of("lon")),
+                List.of(DType.F64, DType.F64),
+                false);
+        DType.Struct inner = new DType.Struct(
+                List.of(ColumnName.of("name"), ColumnName.of("coord")),
+                List.of(DType.UTF8, innermost),
+                false);
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("region"), ColumnName.of("data")),
+                List.of(DType.UTF8, inner),
+                false);
+        try (FileChannel ch = FileChannel.open(vortex, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.cascading(3))) {
+            writer.writeChunk(Map.of(
+                    ColumnName.of("region"), new String[]{"north"},
+                    ColumnName.of("data"), new StructData(List.of(
+                            new String[]{"origin"},
+                            new StructData(List.of(new double[]{1.0}, new double[]{2.0}))))));
+        }
+        Path csv = tmp.resolve("out.csv");
+
+        // When
+        CsvExporter.exportCsv(vortex, csv);
+
+        // Then
+        List<String> result = Files.readAllLines(csv);
+        assertThat(result).containsExactly(
+                "region,data",
+                "north,\"{\"\"name\"\":\"\"origin\"\",\"\"coord\"\":{\"\"lat\"\":1.0,\"\"lon\"\":2.0}}\"");
+    }
 }

writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java

@@ -275,6 +275,9 @@ private static long arrayLength(Object data) {
             case double[] a -> a.length;
             case boolean[] a -> a.length;
             case String[] a -> a.length;
+            // A struct column's row count is its fields' row count (all fields share length,
+            // enforced by StructEncodingEncoder); an empty struct carries no rows.
+            case StructData d -> d.fieldArrays().isEmpty() ? 0L : arrayLength(d.fieldArrays().getFirst());
             case ListData d -> d.outerLen();
             case ListViewData d -> d.outerLen();
             case DateTimePartsData d -> d.timestamps().length;