fix: nullable columns bypass cascade → raw VarBin 10-47x larger than JNI (#258)

MaskedEncodingEncoder.pickInner encoded the non-null values with a fixed
first-match encoder {Primitive, VarBin, FixedSizeList}, so a nullable
low-cardinality string column never reached Dict or FSST — producing files
10–47× larger than the Rust reference on string-heavy Raincloud corpus slugs
(uci-mushroom 22×, uci-online-retail-ii 25×, bi-tablerosistemapenal 47×).

When the EncodeContext carries cascade depth (WriteOptions.cascading(n)),
MaskedEncodingEncoder now passes the non-null values through the same
CascadingCompressor as dense columns. A denseValues() helper substitutes
the empty string for null slots in String[] carriers so Dict/FSST can call
getBytes() without NPE — those slots are masked by the validity bitmap and
never read. Without cascade depth the fixed-fallback path is unchanged.

VortexWriter is also updated to hand masked-column override encoders the
cascade registry and depth, not a depth-0 context.

Results after fix (re-encoded from Raincloud parquet oracle):
  uci-mushroom:       22× → 3.1×
  uci-online-retail-ii: 25× → 3.1×
  bi-tablerosistemapenal: 47× → 6.3×

The remaining gap (~2–6×) is Rust's global per-column dict + tighter chunk
layout and deeper FSST training — a separate concern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: dfa1

CHANGELOG.md

@@ -5,6 +5,12 @@ All notable changes to **vortex-java** are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Fixed
+
+- Nullable Utf8/Binary columns are now compressed through the full cascade instead of stored as raw `vortex.varbin`. `MaskedEncodingEncoder` previously encoded its non-null values with a fixed first-match encoder (primitive/varbin), so a nullable low-cardinality string column never reached Dict or FSST — producing files 10–47× larger than the Rust reference on the string-heavy Raincloud corpus (e.g. `uci-mushroom` 22×, `uci-online-retail-ii` 25×). The masked values now run through the same `CascadingCompressor` as dense columns, dropping those ratios to ~2–3×. ([#258](https://github.com/dfa1/vortex-java/issues/258))
+
 ## [0.12.2] — 2026-07-10
 
 **Registry cleanup and schema fidelity** are the two themes of this release. `ReadRegistry` and

docs/compatibility.md

@@ -95,7 +95,7 @@ Per-slug status lives in `integration/src/test/resources/raincloud/expected-stat
 | `vortex.listview`           | `ListViewEncodingDecoder`        | `ListViewEncodingEncoder`        | ✅      | ✅      |                                                                       |
 | `vortex.fixed_size_list`    | `FixedSizeListEncodingDecoder`   | `FixedSizeListEncodingEncoder`   | ✅      | ✅      |                                                                       |
 | `vortex.zstd`               | `ZstdEncodingDecoder`            | `ZstdEncodingEncoder`            | ✅      | ✅      | Primitive, Utf8, Binary                                               |
-| `vortex.masked`             | `MaskedEncodingDecoder`          | `MaskedEncodingEncoder`          | ✅      | ✅      | NullableData carrier; inner picks Primitive / VarBin / FixedSizeList   |
+| `vortex.masked`             | `MaskedEncodingDecoder`          | `MaskedEncodingEncoder`          | ✅      | ✅      | NullableData carrier; inner values cascade (Dict/FSST/ALP/…) when cascading is enabled, else first-match Primitive / VarBin / FixedSizeList |
 | `vortex.decimal`            | `DecimalEncodingDecoder`         | `DecimalEncodingEncoder`         | ✅      | ✅      |                                                                       |
 | `vortex.decimal_byte_parts` | `DecimalBytePartsEncodingDecoder`| `DecimalBytePartsEncodingEncoder`| ✅      | ✅      |                                                                       |
 | `vortex.datetimeparts`      | `DateTimePartsEncodingDecoder`   | `DateTimePartsEncodingEncoder`   | ✅      | ✅      |                                                                       |

integration/src/test/java/io/github/dfa1/vortex/integration/FileSizeComparisonIntegrationTest.java

@@ -342,6 +342,84 @@ private static void comparePcoI64(Path dir, String name, long[] data,
         assertThat(jniSize).as("JNI pco file for pattern '%s' must be non-empty", name).isGreaterThan(0);
     }
 
+    @Test
+    void nullableLowCardinalityUtf8_javaVsJni(@TempDir Path tmp) throws IOException {
+        // Given — 50k rows of a nullable low-cardinality Utf8 column (8 distinct short strings,
+        // ~10% nulls), mirroring the Raincloud string-heavy corpus (e.g. uci-mushroom). Before
+        // #258 the masked (nullable) path stored the values as raw VarBin — never reaching the
+        // cascade — so the Java file ran 10-47x larger than Rust. Dict must now win on the inner
+        // values, keeping the file small.
+        int n = 50_000;
+        String[] categories = {"alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta"};
+        String[] data = new String[n];
+        java.util.Random rng = new java.util.Random(42);
+        for (int i = 0; i < n; i++) {
+            // Every 10th row is null — exercises the vortex.masked validity wrapper.
+            data[i] = (i % 10 == 0) ? null : categories[rng.nextInt(categories.length)];
+        }
+        DType.Struct javaSchema = new DType.Struct(
+                List.of(ColumnName.of("s")), List.of(DType.UTF8.withNullable(true)), false);
+        Schema jniSchema = new Schema(List.of(
+                Field.nullable("s", new ArrowType.Utf8())));
+
+        // When
+        Path javaFile = tmp.resolve("nullable-lowcard-java.vtx");
+        try (FileChannel ch = FileChannel.open(javaFile, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             VortexWriter writer = VortexWriter.create(
+                     ch, javaSchema, WriteOptions.cascading(3).withGlobalDict(false))) {
+            writer.writeChunk(Map.of(ColumnName.of("s"), data));
+        }
+
+        Path jniFile = tmp.resolve("nullable-lowcard-jni.vtx");
+        String jniUri = jniFile.toAbsolutePath().toUri().toString();
+        try (dev.vortex.api.VortexWriter writer = dev.vortex.api.VortexWriter.create(
+                SESSION, jniUri, jniSchema, new HashMap<>(), ALLOCATOR);
+             VectorSchemaRoot root = VectorSchemaRoot.create(jniSchema, ALLOCATOR)) {
+            VarCharVector vec = (VarCharVector) root.getVector("s");
+            vec.allocateNew();
+            for (int i = 0; i < n; i++) {
+                if (data[i] == null) {
+                    vec.setNull(i);
+                } else {
+                    vec.setSafe(i, data[i].getBytes(StandardCharsets.UTF_8));
+                }
+            }
+            root.setRowCount(n);
+            try (ArrowArray arr = ArrowArray.allocateNew(ALLOCATOR);
+                 ArrowSchema schema = ArrowSchema.allocateNew(ALLOCATOR)) {
+                Data.exportVectorSchemaRoot(ALLOCATOR, root, null, arr, schema);
+                writer.writeBatch(arr.memoryAddress(), schema.memoryAddress());
+            }
+        }
+
+        long javaSize = Files.size(javaFile);
+        long jniSize = Files.size(jniFile);
+        long rawBytes = (long) n * 5;
+        System.out.printf(
+                "[NullableLowCardUtf8] %,d rows  raw=%,d bytes  JNI=%,d bytes  Java=%,d bytes  Java/JNI=%.2fx  Java/raw=%.2fx%n",
+                n, rawBytes, jniSize, javaSize,
+                (double) javaSize / jniSize, (double) javaSize / rawBytes);
+
+        // Then — Java stays well under raw storage: the cascade dict-encodes the masked values
+        // instead of falling through to raw VarBin (the #258 regression, which ran ~20x raw).
+        assertThat(javaSize).as("dict-encoded masked values must beat raw VarBin storage")
+                .isLessThan(rawBytes);
+
+        // Then — Java within 4x of JNI. Rust's per-column dict + tighter chunk layout is still
+        // smaller; the guard's purpose is to catch a regression back to the raw-VarBin masked path.
+        assertThat((double) javaSize / jniSize)
+                .as("nullable low-cardinality Utf8 must not regress to the raw-VarBin masked path")
+                .isLessThan(4.0);
+
+        // Then — Java file is readable, row count and null positions preserved
+        var totalRows = new java.util.concurrent.atomic.AtomicLong();
+        try (VortexReader reader = VortexReader.open(javaFile, ReadRegistry.loadAll());
+             var iter = reader.scan(io.github.dfa1.vortex.reader.ScanOptions.columns("s"))) {
+            iter.forEachRemaining(c -> totalRows.addAndGet(c.column("s").length()));
+        }
+        assertThat(totalRows.get()).isEqualTo(n);
+    }
+
     @Test
     void highCardinalityUtf8_javaVsJni(@TempDir Path tmp) throws IOException {
         // Given — 50k all-distinct short strings. Dict would emit 2-byte codes per row

writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java

@@ -558,7 +558,14 @@ private int writeSegment(DType dtype, Object data, EncodingEncoder encodingOverr
         try (Arena arena = Arena.ofConfined()) {
             EncodeResult result;
             if (encodingOverride != null) {
-                EncodeContext encodeCtx = EncodeContext.of(arena, defaultRegistry);
+                // Give overrides the cascade registry + depth when cascading is enabled, so
+                // wrapping encoders (notably MaskedEncodingEncoder for nullable columns) can
+                // compress their inner values through the full CascadingCompressor rather than a
+                // fixed first-match encoder. Without cascading, a depth-0 context is passed and the
+                // override behaves as before.
+                EncodeContext encodeCtx = options.allowedCascading() > 0
+                        ? EncodeContext.ofDepth(options.allowedCascading(), arena, cascadeRegistry)
+                        : EncodeContext.of(arena, defaultRegistry);
                 result = encodingOverride.encode(dtype, data, encodeCtx);
             } else if (options.allowedCascading() > 0) {
                 EncodeContext encodeCtx = EncodeContext.ofDepth(options.allowedCascading(), arena, cascadeRegistry);

writer/src/main/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoder.java

@@ -11,6 +11,12 @@
 
 /// Write-only encoder for `vortex.masked`. Wraps the payload encode in a values + validity
 /// pair driven by a [NullableData] carrier.
+///
+/// When the context carries cascade depth (positive [EncodeContext#allowedCascading]) the inner
+/// non-nullable values are compressed through the full [CascadingCompressor] built from the
+/// registry, so nullable columns get the same Dict/FSST/ALP/bitpack selection as dense columns.
+/// Without cascade depth the values fall back to a fixed first-match encoder
+/// (primitive / varbin / fixed-size-list).
 public final class MaskedEncodingEncoder implements EncodingEncoder {
 
     private static final List<EncodingEncoder> INNER_FALLBACK = List.of(
@@ -35,8 +41,7 @@ public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
                     "expected NullableData, got " + (data == null ? "null" : data.getClass().getName()));
         }
         DType nonNullable = dtype.withNullable(false);
-        EncodingEncoder inner = pickInner(nonNullable);
-        EncodeResult valuesResult = inner.encode(nonNullable, values, ctx);
+        EncodeResult valuesResult = encodeValues(nonNullable, values, ctx);
         EncodeResult validityResult = new BoolEncodingEncoder().encode(DType.BOOL, validity, ctx);
 
         int valuesBufCount = valuesResult.buffers().size();
@@ -54,6 +59,49 @@ public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
         return new EncodeResult(root, buffers, valuesResult.statsMin(), valuesResult.statsMax());
     }
 
+    /// Encodes the non-null values of a masked column.
+    ///
+    /// With cascade depth available, runs the full [CascadingCompressor] over the registry's
+    /// encoders so low-cardinality strings pick Dict, high-cardinality strings pick FSST, and
+    /// numeric values pick ALP/FOR/bitpacking — the same selection dense columns receive. Without
+    /// cascade depth, falls back to the fixed first-match encoder. Either way [MaskedEncodingEncoder]
+    /// never selects itself: its [#accepts] returns `false`, so the compressor cannot recurse into it.
+    private static EncodeResult encodeValues(DType nonNullable, Object values, EncodeContext ctx) {
+        if (ctx.allowedCascading() > 0) {
+            // The NullableData Utf8/Binary carrier keeps null elements in the String[] at null
+            // positions (validity masks them). Dict/FSST call getBytes() on every element, so
+            // substitute the empty string for nulls first — matching VarBinEncodingEncoder, which
+            // encodes a null as a zero-length slot. Those slots are never read: the enclosing
+            // vortex.masked validity bitmap marks the rows null.
+            Object dense = denseValues(values);
+            List<EncodingEncoder> candidates =
+                    List.copyOf(ctx.registry().encoderMap().values());
+            return new CascadingCompressor(candidates).encode(nonNullable, dense, ctx);
+        }
+        return pickInner(nonNullable).encode(nonNullable, values, ctx);
+    }
+
+    /// Returns `values` unchanged, except a `String[]` with null elements is copied with each null
+    /// replaced by the empty string so cascade encoders (Dict, FSST) can call `getBytes()` safely.
+    ///
+    /// @param values the non-nullable values carrier extracted from the [NullableData]
+    /// @return the same array, or a null-free copy when it is a `String[]` containing nulls
+    private static Object denseValues(Object values) {
+        if (!(values instanceof String[] strings)) {
+            return values;
+        }
+        String[] out = null;
+        for (int i = 0; i < strings.length; i++) {
+            if (strings[i] == null) {
+                if (out == null) {
+                    out = strings.clone();
+                }
+                out[i] = "";
+            }
+        }
+        return out != null ? out : strings;
+    }
+
     private static EncodingEncoder pickInner(DType nonNullable) {
         for (EncodingEncoder e : INNER_FALLBACK) {
             if (e.accepts(nonNullable)) {