perf: vortex.sparse for mixed-validity bitmaps

MaskedEncodingEncoder now tries vortex.sparse (fill = majority
value, patches = minority positions) for a mixed-validity bitmap
and keeps it over the raw vortex.bool bitmap when smaller. The
patch-index array is compressed through a dedicated cascade
(vortex.sequence, fastlanes.delta, FoR, bitpack) kept separate
from VortexWriter's general per-column cascade codec list, so a
clustered or regular null pattern collapses to a handful of bytes
instead of a full n/8-byte bitmap.

Also fixes SequenceEncodingEncoder.encodeCascade: it had no
override and inherited the default that calls encode() directly,
which throws when a stratified sample looks arithmetic but the
full data isn't. That's a pre-existing gap in the general cascade
retry contract (every other encoder returns CascadeStep.notApplicable()
instead of throwing) that the new patch-index cascade path exposed.

On the 200k-row/10-chunk nullable low-cardinality Utf8 benchmark:
107KB -> 86KB, Java/JNI ratio 1.45x -> 1.17x.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: dfa1

CHANGELOG.md

@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Nullable low-cardinality columns now use the shared global dictionary across chunks instead of re-emitting a per-chunk dictionary. The writer previously excluded any nullable column from global-dict candidacy, so a nullable low-cardinality string/numeric column (extremely common in real data — most Raincloud categorical columns have nulls) re-wrote its dictionary in every chunk. Nullable columns now run the same cardinality/ratio gate against their non-null values and emit one shared `vortex.dict` layout with per-chunk masked codes; the reader combines codes-side and pool-side validity per row. On a 200k-row, 10-chunk nullable low-cardinality Utf8 column this shrinks the file 54% (232 KB → 107 KB) and closes the gap to the Rust reference from ~4× to ~1.5×. (internal)
 - `MaskedEncodingEncoder` now encodes an all-valid or all-invalid validity bitmap as `vortex.constant` instead of a raw per-row bitmap. A nullable column chunk with zero nulls previously still paid a full `n/8`-byte bitmap for no information; an all-valid or all-invalid chunk now costs a few bytes regardless of row count. (internal)
+- `MaskedEncodingEncoder` also tries `vortex.sparse` for a mixed-validity bitmap (fill = the majority value, patches = the minority positions) and keeps it over the raw bitmap when smaller — the patch-index array is compressed through a dedicated `vortex.sequence`/`fastlanes.delta`/FoR/bitpack cascade, so a clustered or regular null pattern (e.g. one null every 10 rows) collapses to a handful of bytes instead of a full bitmap. On the 200k-row, 10-chunk nullable low-cardinality Utf8 benchmark this drops the file a further 20% (107 KB → 86 KB) and the gap to the Rust reference from ~1.45× to ~1.17×. Also: `SequenceEncodingEncoder.encodeCascade` no longer lets `encode`'s "not an arithmetic sequence" exception escape when a stratified sample looked arithmetic but the full data wasn't — it now reports the cascade step as not applicable, like every other encoder's contract requires. (internal)
 
 ## [0.12.2] — 2026-07-12
 

TODO.md

@@ -12,14 +12,19 @@
 - [ ] Performance tests must be peer-reviewed
 - [ ] Run performance tests on other machines (I have access only to Apple M5)
 - [ ] **Vector API adoption** — see [ADR-0005](adr/0005-vector-api-adoption.md).
-- [ ] **`DType.Bool` has no RLE/Sparse/Constant encoding** — `RunEndEncodingEncoder`,
-  `RleEncodingEncoder`, `SparseEncodingEncoder`, and `ConstantEncodingEncoder` all `accepts()`
-  only `DType.Primitive`; the sole Bool-capable encoder, `BoolEncodingEncoder`, always emits a raw
-  1-bit/row bitmap. This caps how small a nullable column's `vortex.masked` validity child can get:
-  measured at ~75% of the remaining vortex-jni gap (25 KB of 33 KB) on a 200k-row/10-chunk nullable
-  low-cardinality Utf8 benchmark (`FileSizeComparisonIntegrationTest#nullableLowCardinalityUtf8_multiChunk_globalDict_javaVsJni`,
-  commit 5fe8b544). Needs end-to-end Bool support (write + read) for at least one of RLE/Sparse —
-  comparable effort to the nullable-global-dict fix, not a small tweak.
+- [ ] **`DType.Bool` still has no RLE/Constant-at-the-`accepts()`-level encoding** —
+  `RunEndEncodingEncoder`/`RleEncodingEncoder`/`ConstantEncodingEncoder` all `accepts()` only
+  `DType.Primitive`. `MaskedEncodingEncoder` now covers the all-valid/all-invalid case directly
+  (`vortex.constant`, bypassing `accepts()`) and the mixed-validity case via a dedicated
+  `SparseEncodingEncoder.encodeBool` path (fill = majority value, patches = minority positions,
+  index array cascaded through `vortex.sequence`/`fastlanes.delta`/FoR/bitpack) — closing most of
+  the remaining vortex-jni gap on a clustered/regular null pattern (200k-row/10-chunk nullable
+  low-cardinality Utf8 benchmark: 1.45× → 1.17×, commits 5fe8b544/ecd47ead + Sparse-for-Bool).
+  What's left: a **scattered, non-clustered** null pattern (no exploitable index structure) still
+  costs close to the raw 1-bit/row bitmap — `vortex.runend`/`fastlanes.rle` genuinely support Bool
+  would help there, but that needs the same `accepts()`/fill-value generalization done for
+  `SparseEncodingEncoder`, applied to two more encoders (plus their decoders, if not already
+  Bool-capable like `SparseEncodingDecoder` turned out to be).
 
 ## Security
 

integration/src/test/java/io/github/dfa1/vortex/integration/FileSizeComparisonIntegrationTest.java

@@ -529,12 +529,13 @@ void nullableLowCardinalityUtf8_multiChunk_globalDict_javaVsJni(@TempDir Path tm
                 "[NullableMultiChunkUtf8] %,d rows  %d chunks  JNI=%,d bytes  Java=%,d bytes  Java/JNI=%.2fx%n",
                 n, n / batch, jniSize, javaSize, ratio);
 
-        // Then — global dict across chunks closes most of the gap the per-chunk path leaves. The
-        // bound is set from the measured ratio with headroom; it is tighter than the 4.0x guard on
-        // the single-chunk global-dict-disabled test.
+        // Then — global dict across chunks, plus vortex.sparse on the masked codes' validity
+        // (patch indices further compressed via vortex.sequence/fastlanes.delta on the
+        // clustered/regular null pattern), closes most of the gap the per-chunk path leaves.
+        // The bound is set from the measured ratio (~1.17x) with headroom.
         assertThat(ratio)
                 .as("nullable low-cardinality Utf8 with global dict across chunks stays near JNI")
-                .isLessThan(2.5);
+                .isLessThan(1.5);
 
         // Then — Java file is readable, row count preserved.
         var totalRows = new java.util.concurrent.atomic.AtomicLong();

writer/src/main/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoder.java

@@ -83,9 +83,14 @@ private static EncodeResult encodeValues(DType nonNullable, Object values, Encod
         return pickInner(nonNullable).encode(nonNullable, values, ctx);
     }
 
-    /// Encodes a masked column's validity bitmap, using `vortex.constant` instead of a raw bitmap
-    /// when every row shares the same validity (the whole chunk is all-valid or all-invalid) — a
-    /// common case that otherwise costs a full `n/8`-byte bitmap for no information.
+    /// Encodes a masked column's validity bitmap.
+    ///
+    /// Every row sharing the same validity (the whole chunk all-valid or all-invalid) is
+    /// `vortex.constant` — a common case that otherwise costs a full `n/8`-byte bitmap for no
+    /// information. Otherwise, with cascade depth available, also tries `vortex.sparse`
+    /// ([SparseEncodingEncoder#encodeBool]) — a clustered or regular null pattern can compress its
+    /// patch-index array well below a raw bitmap — and keeps whichever candidate is smaller.
+    /// Without cascade depth, or when sparse doesn't win, falls back to a raw `vortex.bool` bitmap.
     ///
     /// @param validity per-row validity bitmap
     /// @param ctx      the encode context
@@ -96,7 +101,20 @@ private static EncodeResult encodeValidity(boolean[] validity, EncodeContext ctx
             ProtoScalarValue scalar = ProtoScalarValue.ofBoolValue(value);
             return EncodeResult.simple(EncodingId.VORTEX_CONSTANT, MemorySegment.ofArray(scalar.encode()));
         }
-        return new BoolEncodingEncoder().encode(DType.BOOL, validity, ctx);
+        EncodeResult raw = new BoolEncodingEncoder().encode(DType.BOOL, validity, ctx);
+        if (ctx.allowedCascading() <= 0) {
+            return raw;
+        }
+        EncodeResult sparse = SparseEncodingEncoder.encodeBool(validity, ctx);
+        return totalBytes(sparse) < totalBytes(raw) ? sparse : raw;
+    }
+
+    private static long totalBytes(EncodeResult result) {
+        long total = 0;
+        for (MemorySegment buf : result.buffers()) {
+            total += buf.byteSize();
+        }
+        return total;
     }
 
     private static boolean isConstantValidity(boolean[] validity) {

writer/src/main/java/io/github/dfa1/vortex/writer/encode/SequenceEncodingEncoder.java

@@ -23,6 +23,19 @@ public boolean accepts(DType dtype) {
         return dtype instanceof DType.Primitive;
     }
 
+    /// Cascade-aware entry point. [#encode] throws when `data` isn't a perfect arithmetic sequence
+    /// (e.g. a winner selected on a stratified sample that turns out not to hold over the full
+    /// data) — the cascade's retry contract needs a non-applicable [CascadeStep], not an exception,
+    /// so failures are caught here and reported that way instead.
+    @Override
+    public CascadeStep encodeCascade(DType dtype, Object data, EncodeContext ctx) {
+        try {
+            return CascadeStep.terminal(encode(dtype, data, ctx));
+        } catch (VortexException e) {
+            return CascadeStep.notApplicable();
+        }
+    }
+
     @Override
     public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
         if (!(dtype instanceof DType.Primitive p)) {

writer/src/main/java/io/github/dfa1/vortex/writer/encode/SparseEncodingEncoder.java

@@ -16,6 +16,18 @@
 /// Write-only encoder for `vortex.sparse`.
 public final class SparseEncodingEncoder implements EncodingEncoder {
 
+    /// Candidates for compressing a boolean mask's patch-index array ([#encodeBool]) — a sorted,
+    /// often-regular sequence (e.g. a periodic null pattern), so `vortex.sequence`/`fastlanes.delta`
+    /// are tried first. Deliberately separate from [io.github.dfa1.vortex.writer.VortexWriter]'s
+    /// general per-column cascade codec list, which excludes both: adding them there would change
+    /// selection for every dense Primitive column, not just this narrow index-compression case.
+    private static final List<EncodingEncoder> INDEX_CASCADE_CANDIDATES = List.of(
+            new SequenceEncodingEncoder(),
+            new DeltaEncodingEncoder(),
+            new FrameOfReferenceEncodingEncoder(),
+            new BitpackedEncodingEncoder(),
+            new PrimitiveEncodingEncoder());
+
     @Override
     public EncodingId encodingId() {
         return EncodingId.VORTEX_SPARSE;
@@ -184,6 +196,89 @@ private static Object valArr(List<Long> patchBits, PType ptype) {
         };
     }
 
+    /// Encodes a boolean mask as `vortex.sparse`: fill = the majority value, patches = the minority
+    /// positions. Unlike the numeric path (hardcoded fill = 0, `expectedRatio`-gated) this always
+    /// builds the result — a two-valued domain has no "wrong" fill to guess, and the patch index
+    /// array is run through the full [CascadingCompressor] so a clustered or regular null pattern
+    /// (e.g. `fastlanes.delta` on a periodic run of nulls) can beat a raw 1-bit/row bitmap. Callers
+    /// compare the returned size against a raw bitmap and keep whichever is smaller.
+    ///
+    /// @param validity per-row boolean array; must contain at least one `true` and one `false`
+    ///                 (an all-same array is cheaper as `vortex.constant` — see [ConstantEncodingEncoder])
+    /// @param ctx      encode context
+    /// @return the encoded `vortex.sparse` result
+    static EncodeResult encodeBool(boolean[] validity, EncodeContext ctx) {
+        int n = validity.length;
+        int trueCount = 0;
+        for (boolean b : validity) {
+            if (b) {
+                trueCount++;
+            }
+        }
+        boolean fillValue = trueCount * 2 >= n;
+        int numPatches = fillValue ? n - trueCount : trueCount;
+        int[] patchIdx = new int[numPatches];
+        int p = 0;
+        for (int i = 0; i < n; i++) {
+            if (validity[i] != fillValue) {
+                patchIdx[p++] = i;
+            }
+        }
+        boolean[] patchVals = new boolean[numPatches];
+        java.util.Arrays.fill(patchVals, !fillValue);
+
+        PType idxPtype = chooseIdxPtype(n);
+        Object idxArr = idxArr(patchIdx, idxPtype);
+
+        ProtoScalarValue fillScalar = ProtoScalarValue.ofBoolValue(fillValue);
+        byte[] fillBytes = fillScalar.encode();
+        MemorySegment fillBuf = ctx.arena().allocate(fillBytes.length);
+        MemorySegment.copy(MemorySegment.ofArray(fillBytes), 0, fillBuf, 0, fillBytes.length);
+
+        DType idxDtype = new DType.Primitive(idxPtype, false);
+        EncodeResult idxResult = new CascadingCompressor(INDEX_CASCADE_CANDIDATES).encode(idxDtype, idxArr, ctx);
+        EncodeResult valResult = new BoolEncodingEncoder().encode(DType.BOOL, patchVals, ctx);
+
+        List<MemorySegment> buffers = new ArrayList<>();
+        buffers.add(fillBuf);
+        buffers.addAll(idxResult.buffers());
+        int valOffset = 1 + idxResult.buffers().size();
+        buffers.addAll(valResult.buffers());
+
+        EncodeNode idxNode = EncodeNode.remapBufferIndices(idxResult.rootNode(), 1);
+        EncodeNode valNode = EncodeNode.remapBufferIndices(valResult.rootNode(), valOffset);
+
+        ProtoPatchesMetadata patchesMeta = new ProtoPatchesMetadata(
+                numPatches, 0L, io.github.dfa1.vortex.core.proto.ProtoPType.fromValue(idxPtype.ordinal()),
+                null, null, null);
+        byte[] metaBytes = new ProtoSparseMetadata(patchesMeta).encode();
+
+        EncodeNode root = new EncodeNode(EncodingId.VORTEX_SPARSE, MemorySegment.ofArray(metaBytes),
+                new EncodeNode[]{idxNode, valNode}, new int[]{0});
+        return new EncodeResult(root, List.copyOf(buffers), null, null);
+    }
+
+    private static Object idxArr(int[] patchIdx, PType idxPtype) {
+        int n = patchIdx.length;
+        return switch (idxPtype) {
+            case U8 -> {
+                byte[] a = new byte[n];
+                for (int i = 0; i < n; i++) {
+                    a[i] = (byte) patchIdx[i];
+                }
+                yield a;
+            }
+            case U16 -> {
+                short[] a = new short[n];
+                for (int i = 0; i < n; i++) {
+                    a[i] = (short) patchIdx[i];
+                }
+                yield a;
+            }
+            default -> patchIdx;
+        };
+    }
+
     @Override
     public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
         if (!(dtype instanceof DType.Primitive p)) {

writer/src/test/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoderTest.java

@@ -15,6 +15,7 @@
 import io.github.dfa1.vortex.reader.decode.ConstantEncodingDecoder;
 import io.github.dfa1.vortex.reader.decode.MaskedEncodingDecoder;
 import io.github.dfa1.vortex.reader.decode.PrimitiveEncodingDecoder;
+import io.github.dfa1.vortex.writer.WriteRegistry;
 import org.junit.jupiter.api.Test;
 
 import java.lang.foreign.Arena;
@@ -238,4 +239,65 @@ void mixedValidity_stillEncodesAsRawBitmap() {
         assertThat(masked.isValid(1)).isFalse();
         assertThat(masked.isValid(2)).isTrue();
     }
+
+    @Test
+    void withCascade_periodicNulls_prefersSparseOverRawBitmap() {
+        // Given — 2000 rows, every 10th null: a clustered/regular pattern (mirrors the real
+        // low-cardinality-Utf8 benchmark) whose patch-index gaps compress far below a raw bitmap
+        // once cascaded (e.g. fastlanes.delta on a near-constant stride).
+        int n = 2_000;
+        int[] values = new int[n];
+        boolean[] validity = new boolean[n];
+        for (int i = 0; i < n; i++) {
+            values[i] = i;
+            validity[i] = i % 10 != 0;
+        }
+        DType i32Nullable = new DType.Primitive(PType.I32, true);
+        NullableData data = new NullableData(values, validity);
+        EncodeContext cascadeCtx = EncodeContext.ofDepth(3, Arena.ofAuto(), WriteRegistry.loadAll());
+
+        // When
+        EncodeResult result = SUT.encode(i32Nullable, data, cascadeCtx);
+
+        // Then — the validity child picked vortex.sparse over a raw bitmap
+        assertThat(result.rootNode().children()[1].encodingId()).isEqualTo(EncodingId.VORTEX_SPARSE);
+
+        // And every value and null position round-trips exactly
+        Array decoded = DECODER.decode(DecodeTestHelper.toDecodeContext(result, (long) n, i32Nullable, ReadRegistry.loadAll()));
+        MaskedArray masked = (MaskedArray) decoded;
+        for (int i = 0; i < n; i++) {
+            assertThat(masked.isValid(i)).as("row %d", i).isEqualTo(i % 10 != 0);
+        }
+    }
+
+    @Test
+    void withCascade_denseRandomNulls_keepsRawBitmap() {
+        // Given — ~50% nulls with no exploitable structure: sparse patch overhead (index + value
+        // per minority row) can't beat a raw 1-bit/row bitmap, so the comparison must keep the
+        // bitmap rather than always preferring sparse.
+        int n = 2_000;
+        int[] values = new int[n];
+        boolean[] validity = new boolean[n];
+        java.util.Random rng = new java.util.Random(7);
+        for (int i = 0; i < n; i++) {
+            values[i] = i;
+            validity[i] = rng.nextBoolean();
+        }
+        DType i32Nullable = new DType.Primitive(PType.I32, true);
+        NullableData data = new NullableData(values, validity);
+        EncodeContext cascadeCtx = EncodeContext.ofDepth(3, Arena.ofAuto(), WriteRegistry.loadAll());
+
+        // When
+        EncodeResult result = SUT.encode(i32Nullable, data, cascadeCtx);
+
+        // Then
+        assertThat(result.rootNode().children()[1].encodingId()).isEqualTo(EncodingId.VORTEX_BOOL);
+
+        // And it still round-trips exactly
+        Array decoded = DECODER.decode(DecodeTestHelper.toDecodeContext(result, (long) n, i32Nullable, ReadRegistry.loadAll()));
+        MaskedArray masked = (MaskedArray) decoded;
+        for (int i = 0; i < n; i++) {
+            assertThat(masked.isValid(i)).as("row %d", i).isEqualTo(validity[i]);
+        }
+    }
 }