fix: admit nullable columns to global dict

Nullable low-cardinality columns previously bypassed the writer's
shared global dictionary, re-emitting a per-chunk vortex.dict for
every chunk instead of one shared dict for the whole file. Nullable
primitive/Utf8 columns now pass the same cardinality/ratio gate
(skipping null slots) and their per-chunk codes are wrapped in
vortex.masked.

Also fixes two reader gaps this surfaced: DictLayoutDecoder's
Utf8/VarBin branch didn't unwrap masked codes/pool (only the
primitive branch did), and ChunkedLayoutDecoder silently dropped
per-chunk validity when concatenating masked chunks — a pre-existing
bug for any nullable column spanning multiple chunks through that
decode path.

200k-row nullable low-card Utf8 test: 232KB -> 107KB (54% smaller),
Java/JNI ratio 4x -> 1.45x.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: dfa1

CHANGELOG.md

@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- Nullable low-cardinality columns now use the shared global dictionary across chunks instead of re-emitting a per-chunk dictionary. The writer previously excluded any nullable column from global-dict candidacy, so a nullable low-cardinality string/numeric column (extremely common in real data — most Raincloud categorical columns have nulls) re-wrote its dictionary in every chunk. Nullable columns now run the same cardinality/ratio gate against their non-null values and emit one shared `vortex.dict` layout with per-chunk masked codes; the reader combines codes-side and pool-side validity per row. On a 200k-row, 10-chunk nullable low-cardinality Utf8 column this shrinks the file 54% (232 KB → 107 KB) and closes the gap to the Rust reference from ~4× to ~1.5×. (internal)
+
 ## [0.12.2] — 2026-07-12
 
 **Registry cleanup, schema fidelity, and list-column support** are the three themes of this

integration/src/test/java/io/github/dfa1/vortex/integration/FileSizeComparisonIntegrationTest.java

@@ -420,6 +420,146 @@ void nullableLowCardinalityUtf8_javaVsJni(@TempDir Path tmp) throws IOException
         assertThat(totalRows.get()).isEqualTo(n);
     }
 
+    @Test
+    void globalDict_nullableMultiChunkUtf8_smallerThanPerChunkDict(@TempDir Path tmp) throws IOException {
+        // Given — a NULLABLE low-cardinality Utf8 column (8 categories, ~10% nulls) spanning several
+        // chunks. Per-chunk-dict mode re-emits the same 8-string dictionary in every chunk; global
+        // mode emits it once, sharing per-chunk masked codes. This is the direct regression guard for
+        // admitting nullable columns to the global dict.
+        int rows = 200_000;
+        int batch = 20_000;
+        String[] categories = {"alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta"};
+        java.util.Random rng = new java.util.Random(42);
+        String[][] chunks = new String[rows / batch][];
+        for (int c = 0; c < chunks.length; c++) {
+            String[] chunk = new String[batch];
+            for (int i = 0; i < batch; i++) {
+                int r = c * batch + i;
+                chunk[i] = (r % 10 == 0) ? null : categories[rng.nextInt(categories.length)];
+            }
+            chunks[c] = chunk;
+        }
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("s")), List.of(DType.UTF8.withNullable(true)), false);
+
+        // When
+        Path globalDictFile = writeNullableUtf8Chunks(tmp, "gdict-on.vtx", schema, chunks, true);
+        Path perChunkDictFile = writeNullableUtf8Chunks(tmp, "gdict-off.vtx", schema, chunks, false);
+        long globalDictSize = Files.size(globalDictFile);
+        long perChunkDictSize = Files.size(perChunkDictFile);
+
+        // Then — report
+        System.out.printf(
+                "[GlobalDictNullableUtf8] %,d rows  %d chunks  globalDict=%,d bytes  perChunkDict=%,d bytes  saving=%.2f%%%n",
+                rows, chunks.length, globalDictSize, perChunkDictSize,
+                100.0 * (perChunkDictSize - globalDictSize) / perChunkDictSize);
+
+        // Then — global dict is strictly smaller than the per-chunk-dict baseline for identical
+        // nullable data.
+        assertThat(globalDictSize).isLessThan(perChunkDictSize);
+
+        // Then — global dict file readable, row count matches.
+        var totalRows = new java.util.concurrent.atomic.AtomicLong();
+        try (VortexReader reader = VortexReader.open(globalDictFile, ReadRegistry.loadAll());
+             var iter = reader.scan(io.github.dfa1.vortex.reader.ScanOptions.columns("s"))) {
+            iter.forEachRemaining(c -> totalRows.addAndGet(c.column("s").length()));
+        }
+        assertThat(totalRows.get()).isEqualTo(rows);
+    }
+
+    @Test
+    void nullableLowCardinalityUtf8_multiChunk_globalDict_javaVsJni(@TempDir Path tmp) throws IOException {
+        // Given — the same nullable low-cardinality column as nullableLowCardinalityUtf8_javaVsJni,
+        // but written across MULTIPLE chunks with global dict ENABLED (default WriteOptions). Global
+        // dict only pays off across chunks (a single chunk sees no benefit over per-chunk dict), so
+        // this exercises the shared-dictionary path the fix adds and measures its ratio against JNI.
+        int n = 200_000;
+        int batch = 20_000;
+        String[] categories = {"alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta"};
+        String[] data = new String[n];
+        java.util.Random rng = new java.util.Random(42);
+        for (int i = 0; i < n; i++) {
+            data[i] = (i % 10 == 0) ? null : categories[rng.nextInt(categories.length)];
+        }
+        DType.Struct javaSchema = new DType.Struct(
+                List.of(ColumnName.of("s")), List.of(DType.UTF8.withNullable(true)), false);
+        Schema jniSchema = new Schema(List.of(
+                Field.nullable("s", new ArrowType.Utf8())));
+
+        // When — Java: multiple chunks, global dict on (default cascading(3)).
+        Path javaFile = tmp.resolve("nullable-multichunk-java.vtx");
+        try (FileChannel ch = FileChannel.open(javaFile, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             VortexWriter writer = VortexWriter.create(ch, javaSchema, WriteOptions.cascading(3))) {
+            for (int off = 0; off < n; off += batch) {
+                String[] chunk = new String[batch];
+                System.arraycopy(data, off, chunk, 0, batch);
+                writer.writeChunk(Map.of(ColumnName.of("s"), chunk));
+            }
+        }
+
+        Path jniFile = tmp.resolve("nullable-multichunk-jni.vtx");
+        String jniUri = jniFile.toAbsolutePath().toUri().toString();
+        try (dev.vortex.api.VortexWriter writer = dev.vortex.api.VortexWriter.create(
+                SESSION, jniUri, jniSchema, new HashMap<>(), ALLOCATOR);
+             VectorSchemaRoot root = VectorSchemaRoot.create(jniSchema, ALLOCATOR)) {
+            for (int off = 0; off < n; off += batch) {
+                VarCharVector vec = (VarCharVector) root.getVector("s");
+                vec.reset();
+                vec.allocateNew();
+                for (int i = 0; i < batch; i++) {
+                    if (data[off + i] == null) {
+                        vec.setNull(i);
+                    } else {
+                        vec.setSafe(i, data[off + i].getBytes(StandardCharsets.UTF_8));
+                    }
+                }
+                root.setRowCount(batch);
+                try (ArrowArray arr = ArrowArray.allocateNew(ALLOCATOR);
+                     ArrowSchema schema = ArrowSchema.allocateNew(ALLOCATOR)) {
+                    Data.exportVectorSchemaRoot(ALLOCATOR, root, null, arr, schema);
+                    writer.writeBatch(arr.memoryAddress(), schema.memoryAddress());
+                }
+            }
+        }
+
+        long javaSize = Files.size(javaFile);
+        long jniSize = Files.size(jniFile);
+        double ratio = (double) javaSize / jniSize;
+        System.out.printf(
+                "[NullableMultiChunkUtf8] %,d rows  %d chunks  JNI=%,d bytes  Java=%,d bytes  Java/JNI=%.2fx%n",
+                n, n / batch, jniSize, javaSize, ratio);
+
+        // Then — global dict across chunks closes most of the gap the per-chunk path leaves. The
+        // bound is set from the measured ratio with headroom; it is tighter than the 4.0x guard on
+        // the single-chunk global-dict-disabled test.
+        assertThat(ratio)
+                .as("nullable low-cardinality Utf8 with global dict across chunks stays near JNI")
+                .isLessThan(2.5);
+
+        // Then — Java file is readable, row count preserved.
+        var totalRows = new java.util.concurrent.atomic.AtomicLong();
+        try (VortexReader reader = VortexReader.open(javaFile, ReadRegistry.loadAll());
+             var iter = reader.scan(io.github.dfa1.vortex.reader.ScanOptions.columns("s"))) {
+            iter.forEachRemaining(c -> totalRows.addAndGet(c.column("s").length()));
+        }
+        assertThat(totalRows.get()).isEqualTo(n);
+    }
+
+    /// Writes a nullable Utf8 column across the given per-chunk arrays, with the global-dict option
+    /// toggled. Returns the file path.
+    private static Path writeNullableUtf8Chunks(Path dir, String filename, DType.Struct schema,
+            String[][] chunks, boolean globalDict) throws IOException {
+        Path file = dir.resolve(filename);
+        try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             VortexWriter writer = VortexWriter.create(
+                     ch, schema, WriteOptions.cascading(3).withGlobalDict(globalDict))) {
+            for (String[] chunk : chunks) {
+                writer.writeChunk(Map.of(ColumnName.of("s"), chunk));
+            }
+        }
+        return file;
+    }
+
     @Test
     void highCardinalityUtf8_javaVsJni(@TempDir Path tmp) throws IOException {
         // Given — 50k all-distinct short strings. Dict would emit 2-byte codes per row

reader/src/main/java/io/github/dfa1/vortex/reader/layout/ChunkedLayoutDecoder.java

@@ -13,8 +13,13 @@
 import io.github.dfa1.vortex.reader.array.ChunkedIntArray;
 import io.github.dfa1.vortex.reader.array.ChunkedLongArray;
 import io.github.dfa1.vortex.reader.array.ChunkedShortArray;
+import io.github.dfa1.vortex.reader.array.BoolArray;
+import io.github.dfa1.vortex.reader.array.MaskedArray;
+import io.github.dfa1.vortex.reader.array.MaterializedBoolArray;
 import io.github.dfa1.vortex.reader.array.VarBinArray;
 
+import java.lang.foreign.MemorySegment;
+import java.lang.foreign.SegmentAllocator;
 import java.lang.foreign.ValueLayout;
 import java.util.ArrayList;
 import java.util.List;
@@ -36,6 +41,46 @@ public Array decode(LayoutDecodeContext ctx, Layout layout, DType dtype) {
         return decodeChunkedLayout(ctx, flats, dtype, layout.rowCount());
     }
 
+    /// Concatenates the per-chunk validity of masked chunks into one row-level bitmap, or returns
+    /// `null` when no chunk is nullable. Unmasked chunks (and masked chunks with a `null` validity,
+    /// which mean all-valid) contribute all-valid rows. This preserves nullability that the
+    /// `ChunkedXxxArray.of` constructors deliberately drop when flattening masked chunks — needed so
+    /// a nullable chunked column (e.g. the codes of a nullable global-dict column) keeps its nulls.
+    ///
+    /// @param chunkArrays the decoded per-chunk arrays, in row order
+    /// @param totalRows    the total logical row count across all chunks
+    /// @param arena        allocator for the combined bitmap
+    /// @return a bit-packed row-validity [BoolArray] of `totalRows` bits, or `null` if all valid
+    private static BoolArray combineChunkValidity(List<Array> chunkArrays, long totalRows,
+            SegmentAllocator arena) {
+        boolean anyMasked = false;
+        for (Array chunk : chunkArrays) {
+            if (chunk instanceof MaskedArray m && m.validity() != null) {
+                anyMasked = true;
+                break;
+            }
+        }
+        if (!anyMasked) {
+            return null;
+        }
+        MemorySegment bits = arena.allocate((totalRows + 7) >>> 3);
+        long row = 0;
+        for (Array chunk : chunkArrays) {
+            long chunkLen = chunk.length();
+            BoolArray validity = chunk instanceof MaskedArray m ? m.validity() : null;
+            for (long i = 0; i < chunkLen; i++) {
+                if (validity == null || validity.getBoolean(i)) {
+                    long globalRow = row + i;
+                    long byteIdx = globalRow >>> 3;
+                    byte cur = bits.get(ValueLayout.JAVA_BYTE, byteIdx);
+                    bits.set(ValueLayout.JAVA_BYTE, byteIdx, (byte) ((cur & 0xff) | (1 << (globalRow & 7))));
+                }
+            }
+            row += chunkLen;
+        }
+        return new MaterializedBoolArray(DType.BOOL, totalRows, bits.asReadOnly());
+    }
+
     /// Flattens a layout subtree into its ordered flat (and dict) leaves. A private copy of
     /// `ScanIterator.collectFlats`: the scan keeps its own for chunk-shape planning, while the
     /// chunked decoder needs the same flattening to build the leaf array list.
@@ -80,21 +125,28 @@ private static Array decodeChunkedLayout(LayoutDecodeContext ctx, List<Layout> f
             // a leaf's layout id must be honored under a chunked parent too.
             chunkArrays.add(ctx.decodeChild(flat, dtype));
         }
+        Array data;
         if (dtype instanceof DType.Bool) {
-            return ChunkedBoolArray.of(dtype, totalRows, chunkArrays);
-        }
-        if (dtype instanceof DType.Utf8 || dtype instanceof DType.Binary) {
-            return VarBinArray.ChunkedMode.of(dtype, totalRows, chunkArrays);
+            data = ChunkedBoolArray.of(dtype, totalRows, chunkArrays);
+        } else if (dtype instanceof DType.Utf8 || dtype instanceof DType.Binary) {
+            data = VarBinArray.ChunkedMode.of(dtype, totalRows, chunkArrays);
+        } else {
+            PType ptype = ((DType.Primitive) dtype).ptype();
+            data = switch (ptype) {
+                case I64, U64 -> ChunkedLongArray.of(dtype, totalRows, chunkArrays);
+                case I32, U32 -> ChunkedIntArray.of(dtype, totalRows, chunkArrays);
+                case F64 -> ChunkedDoubleArray.of(dtype, totalRows, chunkArrays);
+                case F32 -> ChunkedFloatArray.of(dtype, totalRows, chunkArrays);
+                case I16, U16 -> ChunkedShortArray.of(dtype, totalRows, chunkArrays);
+                case I8, U8 -> ChunkedByteArray.of(dtype, totalRows, chunkArrays);
+                default -> throw new VortexException("unsupported ptype for chunked layout: " + ptype);
+            };
         }
-        PType ptype = ((DType.Primitive) dtype).ptype();
-        return switch (ptype) {
-            case I64, U64 -> ChunkedLongArray.of(dtype, totalRows, chunkArrays);
-            case I32, U32 -> ChunkedIntArray.of(dtype, totalRows, chunkArrays);
-            case F64 -> ChunkedDoubleArray.of(dtype, totalRows, chunkArrays);
-            case F32 -> ChunkedFloatArray.of(dtype, totalRows, chunkArrays);
-            case I16, U16 -> ChunkedShortArray.of(dtype, totalRows, chunkArrays);
-            case I8, U8 -> ChunkedByteArray.of(dtype, totalRows, chunkArrays);
-            default -> throw new VortexException("unsupported ptype for chunked layout: " + ptype);
-        };
+        // The ChunkedXxxArray.of constructors flatten masked chunks to their inner data, dropping
+        // validity. Recover per-chunk nullability by concatenating chunk validity into one bitmap and
+        // re-wrapping — otherwise a nullable chunked column (e.g. nullable global-dict codes) loses
+        // its nulls when it spans more than one chunk.
+        BoolArray validity = combineChunkValidity(chunkArrays, totalRows, ctx.arena());
+        return validity != null ? new MaskedArray(data, validity) : data;
     }
 }

reader/src/main/java/io/github/dfa1/vortex/reader/layout/DictLayoutDecoder.java

@@ -65,23 +65,38 @@ public Array decode(LayoutDecodeContext ctx, Layout dictLayout, DType dtype) {
 
         // VarBin (string) dict: VarBinArray is a sealed interface; ofDict returns the
         // lazy DictMode record (no eager expansion into per-row offsets/bytes).
-        if (values instanceof VarBinArray.OffsetMode vb) {
+        // Unwrap a masked (nullable) codes/values child so the string expansion sees the raw
+        // payload; the row-level validity is re-applied by wrapping the result below. This mirrors
+        // the primitive path (buildLazyDictPrimitive) and is the shape a nullable global-dict Utf8
+        // column produces (masked codes + non-nullable pool).
+        BoolArray poolValidity = values instanceof MaskedArray mv ? mv.validity() : null;
+        Array valuesData = values instanceof MaskedArray mv ? mv.inner() : values;
+        BoolArray codesValidity = codes instanceof MaskedArray mc ? mc.validity() : null;
+        Array codesData = codes instanceof MaskedArray mc ? mc.inner() : codes;
+        if (valuesData instanceof VarBinArray.OffsetMode vb) {
             // Zip-bomb guard: read the codes as a segment so we can validate the buffer
             // before allocating the expansion output. For direct-mapped encodings (e.g.
             // vortex.primitive), the codes buffer is mmap-bounded and can be much smaller
             // than the claimed rowCount. Full-decode encodings (e.g. bitpacked) already
             // wrote n * elemBytes to the arena during decodeChild above, so their buffer
             // matches n.
-            MemorySegment codesSeg = codes.materialize(arena);
+            MemorySegment codesSeg = codesData.materialize(arena);
             long bufferCodes = codesSeg.byteSize() / codesPType.byteSize();
             if (bufferCodes < n) {
                 throw new VortexException(EncodingId.VORTEX_DICT,
                         "dict codes: layout row_count=" + n + " exceeds buffer capacity=" + bufferCodes);
             }
             MemorySegment valOffsets = vb.offsetsSegment();
             PType valOffPType = vb.offsetsPtype();
-            return VarBinArray.ofDict(dtype, n, vb.bytesSegment(), valOffsets, valOffPType,
+            Array dict = VarBinArray.ofDict(dtype, n, vb.bytesSegment(), valOffsets, valOffPType,
                     codesSeg, codesPType);
+            if (poolValidity == null && codesValidity == null) {
+                return dict;
+            }
+            if (poolValidity == null) {
+                return new MaskedArray(dict, codesValidity);
+            }
+            return new MaskedArray(dict, gatherRowValidity(codesData, codesValidity, poolValidity, n, arena));
         }
         if (dtype instanceof DType.Primitive pDtype) {
             // Zip-bomb guard (lazy path): the codes Array has already been decoded above;