fix: propagate row validity through wrapper decoders; NullArray CSV export

Files from the Python bindings (vortex-data 0.69.0) carry row validity
deep in the encoding tree, in three representations our decoders
flattened away — every one silent corruption (nulls exported as
invented values), found by the Raincloud conformance suite (#205):

- a trailing bool validity child on fastlanes.bitpacked, reached
  through vortex.alp / vortex.zigzag / fastlanes.for, which per the
  Rust ValidityChild contract delegate their validity to the encoded
  child. AlpEncodingDecoder and ZigZagEncodingDecoder now decode that
  child as an Array (nullability inherited from the parent dtype),
  split off a MaskedArray, and re-wrap their result — the pattern
  FrameOfReferenceEncodingDecoder already used. Bitpacked decodes the
  trailing validity child at the index its patch shape dictates
  (0 / 2 / 3, mirroring the Rust vtable deserialize).
- dict pools with invalid slots: null rows are codes pointing at an
  invalid pool entry; slot validity is now gathered through the codes
  into per-row validity.
- dict codes with their own validity child: propagated directly, and
  combined with pool validity when both are present. Applied to the
  eager vortex.dict decoder (primitive + utf8 paths) and the lazy dict
  layout path.

Also: CSV export renders all-null (DType.Null) columns as empty fields
instead of throwing (kepler has fully-empty columns).

Corpus evidence: penguins bill_length/bill_depth and 20k+ kepler cells
exported values where the file holds nulls (verified against both the
Parquet oracle and the Python bindings reading the same file). After
the fix kepler passes value-for-value on all 153 columns and flips to
ok in the conformance matrix; penguins' remaining failure is #208 only.

Closes #210
Closes #211

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: dfa1

CHANGELOG.md

@@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 
 - Lazy dict decode now covers I8/U8/I16/U16 value columns (`DictByteArray`, `DictShortArray`) — files from the Python bindings dict-encode narrow-integer columns, which previously failed to scan with `unsupported ptype for lazy dict`. ([#206](https://github.com/dfa1/vortex-java/issues/206))
+- CSV export handles all-null (`DType.Null`) columns as empty fields instead of throwing `unsupported array type: NullArray`. ([#211](https://github.com/dfa1/vortex-java/issues/211))
+- Null rows no longer silently decode as values: wrapper decoders now propagate the row validity that files from the Python bindings carry deep in the encoding tree. Three representations were dropped — a trailing validity child on `fastlanes.bitpacked` (reached through `vortex.alp`/`vortex.zigzag`/`fastlanes.for`, which delegate validity to their encoded child per the Rust `ValidityChild` contract), dict pools with invalid slots, and dict codes with their own validity — across both the eager `vortex.dict` decoder and the lazy dict layout path. Found on real data: penguins and kepler exported invented values (`32.1`, `0.0`) for thousands of null cells. ([#210](https://github.com/dfa1/vortex-java/issues/210))
 
 ### Added
 

csv/src/main/java/io/github/dfa1/vortex/csv/CsvExporter.java

@@ -13,6 +13,7 @@
 import io.github.dfa1.vortex.reader.array.LongArray;
 import io.github.dfa1.vortex.reader.array.ShortArray;
 import io.github.dfa1.vortex.reader.array.MaskedArray;
+import io.github.dfa1.vortex.reader.array.NullArray;
 import io.github.dfa1.vortex.reader.array.VarBinArray;
 import io.github.dfa1.vortex.reader.VortexReader;
 import io.github.dfa1.vortex.reader.ScanIterator;
@@ -151,6 +152,9 @@ private static String cellValue(Array arr, long rowIdx) {
             // Nullable columns decode as MaskedArray: null rows export as an empty field, valid
             // rows defer to the inner values array.
             case MaskedArray ma -> ma.isValid(rowIdx) ? cellValue(ma.inner(), rowIdx) : "";
+            // All-null columns (DType.Null) hold only a row count: every cell is an empty
+            // field, same rule as a MaskedArray null row.
+            case NullArray ignored -> "";
             default -> throw new VortexException(
                     "unsupported array type for CSV export: " + arr.getClass().getSimpleName());
         };

csv/src/test/java/io/github/dfa1/vortex/csv/CsvExporterTest.java

@@ -4,6 +4,8 @@
 import io.github.dfa1.vortex.core.model.DType;
 import io.github.dfa1.vortex.writer.VortexWriter;
 import io.github.dfa1.vortex.writer.WriteOptions;
+import io.github.dfa1.vortex.writer.encode.NullEncodingEncoder;
+import io.github.dfa1.vortex.writer.encode.PrimitiveEncodingEncoder;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.io.TempDir;
 
@@ -69,6 +71,32 @@ void exportsToWriter(@TempDir Path tmp) throws Exception {
         assertThat(lines[2]).isEqualTo("2.7");
     }
 
+    @Test
+    void exportsAllNullColumnAsEmptyFields(@TempDir Path tmp) throws Exception {
+        // Given a file mixing a value column with an all-null (DType.Null) column —
+        // the shape that made real-world exports throw (#211); the long[] carrier for
+        // the null column only supplies the row count
+        Path vortex = tmp.resolve("data.vortex");
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("id"), ColumnName.of("empty")),
+                List.of(DType.I64, DType.NULL),
+                false);
+        try (FileChannel ch = FileChannel.open(vortex, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             // explicit encoder list: the default cascading set has no DType.Null codec
+             VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.defaults(),
+                     List.of(new NullEncodingEncoder(), new PrimitiveEncodingEncoder()))) {
+            writer.writeChunk(Map.of(ColumnName.of("id"), new long[]{1L, 2L}, ColumnName.of("empty"), new long[2]));
+        }
+        Path csv = tmp.resolve("out.csv");
+
+        // When
+        CsvExporter.exportCsv(vortex, csv);
+
+        // Then — null cells are empty fields, same as MaskedArray null rows
+        List<String> result = Files.readAllLines(csv);
+        assertThat(result).containsExactly("id,empty", "1,", "2,");
+    }
+
     @Test
     void suppressesHeaderWhenConfigured(@TempDir Path tmp) throws Exception {
         // Given

docs/compatibility.md

@@ -62,14 +62,15 @@ Per-slug status lives in `integration/src/test/resources/raincloud/expected-stat
 (`ok` must pass; `gap:<issue>` must still fail, so a fix flips the entry in the same change;
 `untriaged` runs and reports without failing the build). A scheduled workflow
 (`raincloud-conformance.yml`) hydrates a size-capped subset weekly. Current triage —
-5 `ok`, 5 known gaps: nested struct columns in scan
+6 `ok`, 4 known gaps: nested struct columns in scan
 ([#207](https://github.com/dfa1/vortex-java/issues/207)), unsigned integers rendered signed
 ([#208](https://github.com/dfa1/vortex-java/issues/208) — silent corruption), RLE over F64
-([#209](https://github.com/dfa1/vortex-java/issues/209)), nullable ALP validity dropped
-([#210](https://github.com/dfa1/vortex-java/issues/210) — silent corruption), all-null
-columns in CSV export ([#211](https://github.com/dfa1/vortex-java/issues/211));
-237 slugs untriaged. Fixed so far by this suite: lazy dict U8/U16 values
-([#206](https://github.com/dfa1/vortex-java/issues/206)).
+([#209](https://github.com/dfa1/vortex-java/issues/209)); 237 slugs untriaged. Fixed so
+far by this suite: lazy dict U8/U16 values
+([#206](https://github.com/dfa1/vortex-java/issues/206)), row validity dropped by wrapper
+decoders ([#210](https://github.com/dfa1/vortex-java/issues/210) — silent corruption),
+all-null columns in CSV export
+([#211](https://github.com/dfa1/vortex-java/issues/211)).
 
 ## Encodings
 

integration/src/test/resources/raincloud/expected-status.csv

@@ -124,7 +124,7 @@ humaneval,untriaged
 insurance,untriaged
 international-football-results-from-1872-to-2017,untriaged
 iowa-liquor-sales,untriaged
-kepler-exoplanet-search-results,gap:211
+kepler-exoplanet-search-results,ok
 kepler-labelled-time-series-data,untriaged
 laion-400m,untriaged
 librispeech-test-clean,untriaged
@@ -166,7 +166,7 @@ osmi-mental-health-in-tech-2020,untriaged
 osmi-mental-health-in-tech-2021,untriaged
 osmi-mental-health-in-tech-2022,untriaged
 osmi-mental-health-in-tech-2023,untriaged
-palmer-archipelago-antarctica-penguin-data,gap:210
+palmer-archipelago-antarctica-penguin-data,gap:208
 peoples-speech-clean-validation,untriaged
 personal-key-indicators-of-heart-disease,untriaged
 pleias-synth,untriaged

reader/src/main/java/io/github/dfa1/vortex/reader/decode/AlpEncodingDecoder.java

@@ -8,10 +8,12 @@
 import io.github.dfa1.vortex.core.proto.ProtoALPMetadata;
 import io.github.dfa1.vortex.core.proto.ProtoPatchesMetadata;
 import io.github.dfa1.vortex.reader.array.Array;
+import io.github.dfa1.vortex.reader.array.BoolArray;
 import io.github.dfa1.vortex.reader.array.LazyAlpDoubleArray;
 import io.github.dfa1.vortex.reader.array.LazyAlpFloatArray;
 import io.github.dfa1.vortex.reader.array.LazyConstantDoubleArray;
 import io.github.dfa1.vortex.reader.array.LazyConstantFloatArray;
+import io.github.dfa1.vortex.reader.array.MaskedArray;
 import io.github.dfa1.vortex.reader.array.MaterializedDoubleArray;
 import io.github.dfa1.vortex.reader.array.MaterializedFloatArray;
 
@@ -55,21 +57,38 @@ public Array decode(DecodeContext ctx) {
         PType ptype = p.ptype();
         long n = ctx.rowCount();
 
-        return switch (ptype) {
-            case F64 -> decodeF64(ctx, meta, expE, expF, n);
-            case F32 -> decodeF32(ctx, meta, expE, expF, n);
+        // Validity mirrors the Rust reference (`ValidityChild<ALP>`): an ALP array's
+        // validity IS its encoded child's validity, and the encoded child's dtype
+        // inherits the ALP dtype's nullability. Decode the child as an Array so a
+        // nullable primitive surfaces its MaskedArray instead of being flattened to a
+        // raw segment (which silently dropped nulls — #210).
+        DType.Primitive encodedDtype = new DType.Primitive(
+                ptype == PType.F64 ? PType.I64 : PType.I32, p.nullable());
+        Array encoded = ctx.decodeChild(0, encodedDtype, n);
+        BoolArray validity = null;
+        Array rawEncoded = encoded;
+        if (encoded instanceof MaskedArray masked) {
+            rawEncoded = masked.inner();
+            validity = masked.validity();
+        }
+        MemorySegment src = ctx.materialize(rawEncoded);
+
+        Array result = switch (ptype) {
+            case F64 -> decodeF64(ctx, meta, expE, expF, n, src);
+            case F32 -> decodeF32(ctx, meta, expE, expF, n, src);
             default -> throw new VortexException(EncodingId.VORTEX_ALP, "unsupported dtype " + ptype);
         };
+        return validity != null ? new MaskedArray(result, validity) : result;
     }
 
-    private static Array decodeF64(DecodeContext ctx, ProtoALPMetadata meta, int expE, int expF, long n) {
+    private static Array decodeF64(DecodeContext ctx, ProtoALPMetadata meta, int expE, int expF, long n,
+            MemorySegment src) {
         // Decode formula mirrors the Rust reference (`ALPFloat::decode_single`): two-step
         // `encoded * F10[f] * IF10[e]`. A pre-multiplied `scale = F10[f] * IF10[e]`
         // gives different IEEE rounding for non-trivial `expF`, breaking round-trip with
         // the encoder's verify step.
         double df = F10_F64[expF];
         double de = IF10_F64[expE];
-        MemorySegment src = ctx.decodeChildSegment(0, DType.I64, n);
         long srcCap = SegmentBroadcast.capacity(src, 8);
 
         if (meta.patches() == null) {
@@ -96,10 +115,10 @@ private static Array decodeF64(DecodeContext ctx, ProtoALPMetadata meta, int exp
         return new MaterializedDoubleArray(ctx.dtype(), n, buf.asReadOnly());
     }
 
-    private static Array decodeF32(DecodeContext ctx, ProtoALPMetadata meta, int expE, int expF, long n) {
+    private static Array decodeF32(DecodeContext ctx, ProtoALPMetadata meta, int expE, int expF, long n,
+            MemorySegment src) {
         float df = F10_F32[expF];
         float de = IF10_F32[expE];
-        MemorySegment src = ctx.decodeChildSegment(0, DType.I32, n);
         long srcCap = SegmentBroadcast.capacity(src, 4);
 
         if (meta.patches() == null) {

reader/src/main/java/io/github/dfa1/vortex/reader/decode/BitpackedEncodingDecoder.java

@@ -8,6 +8,8 @@
 import io.github.dfa1.vortex.core.proto.ProtoBitPackedMetadata;
 import io.github.dfa1.vortex.core.proto.ProtoPatchesMetadata;
 import io.github.dfa1.vortex.reader.array.Array;
+import io.github.dfa1.vortex.reader.array.BoolArray;
+import io.github.dfa1.vortex.reader.array.MaskedArray;
 import io.github.dfa1.vortex.reader.array.MaterializedByteArray;
 import io.github.dfa1.vortex.reader.array.MaterializedIntArray;
 import io.github.dfa1.vortex.reader.array.MaterializedLongArray;
@@ -59,13 +61,50 @@ public Array decode(DecodeContext ctx) {
             applyPatches(ctx, meta.patches(), output, ptype.byteSize());
         }
 
-        return switch (ptype) {
+        Array values = switch (ptype) {
             case I64, U64 -> new MaterializedLongArray(ctx.dtype(), rowCount, output);
             case I32, U32 -> new MaterializedIntArray(ctx.dtype(), rowCount, output);
             case I16, U16 -> new MaterializedShortArray(ctx.dtype(), rowCount, output);
             case I8, U8 -> new MaterializedByteArray(ctx.dtype(), rowCount, output);
             default -> throw new VortexException(EncodingId.FASTLANES_BITPACKED, "unsupported ptype " + ptype);
         };
+        return wrapValidity(ctx, meta, values, rowCount);
+    }
+
+    /// Row validity mirrors the Rust reference (`BitPacked` vtable `deserialize`): the
+    /// children are `[patch_indices, patch_values, patch_chunk_offsets?]` (present only
+    /// with patches) followed by an optional trailing bool validity child. Encodings
+    /// stacked above bitpacked (ALP, FoR, …) delegate their validity here
+    /// (`ValidityChild`), so dropping this child silently un-nulls rows (#210).
+    ///
+    /// @param ctx      decode context
+    /// @param meta     bitpacked metadata (patch shape determines the validity index)
+    /// @param values   the unpacked (and patched) values array
+    /// @param rowCount logical row count
+    /// @return `values`, wrapped in a [MaskedArray] when a validity child is present
+    private static Array wrapValidity(DecodeContext ctx, ProtoBitPackedMetadata meta, Array values, long rowCount) {
+        int validityIdx = validityChildIndex(meta);
+        int childCount = ctx.node().children().length;
+        if (childCount == validityIdx) {
+            return values;
+        }
+        if (childCount != validityIdx + 1) {
+            throw new VortexException(EncodingId.FASTLANES_BITPACKED,
+                    "expected " + validityIdx + " or " + (validityIdx + 1) + " children, got " + childCount);
+        }
+        Array va = ctx.decodeChild(validityIdx, DType.BOOL, rowCount);
+        if (!(va instanceof BoolArray validity)) {
+            throw new VortexException(EncodingId.FASTLANES_BITPACKED,
+                    "validity child decoded to unexpected type: " + va.getClass().getSimpleName());
+        }
+        return new MaskedArray(values, validity);
+    }
+
+    private static int validityChildIndex(ProtoBitPackedMetadata meta) {
+        if (meta.patches() == null) {
+            return 0;
+        }
+        return meta.patches().chunk_offsets_ptype() != null ? 3 : 2;
     }
 
     private static void fastlanesUnpackToSeg(