perf: vortex.runend and Bool-capable vortex.constant

dfa1 · claude · dfa1 · commit 5379af66f67d · 2026-07-12T18:47:33.000+02:00
RunEndEncodingEncoder gains an encodeBool path (consecutive equal
values collapse into runs) — complements vortex.sparse by winning
on long clustered runs of valid/invalid rows instead of a dominant
value with scattered flips. MaskedEncodingEncoder now tries both
plus a raw bitmap and keeps whichever is smallest.

ConstantEncodingEncoder now accepts DType.Bool at the accepts()
level (previously Primitive only), so any all-true/all-false dense
Bool column can win vortex.constant through the normal per-column
cascade, not only via MaskedEncodingEncoder's dedicated check.
MaskedEncodingEncoder's validity encoding now delegates to it
instead of duplicating the scalar-building logic.

Both reader decoders (RunEndEncodingDecoder, ConstantEncodingDecoder)
already handled DType.Bool before this change — writer-only fix,
same pattern as the vortex.sparse work.

Co-Authored-By: Claude Sonnet 5 &lt;noreply@anthropic.com&gt;
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Nullable low-cardinality columns now use the shared global dictionary across chunks instead of re-emitting a per-chunk dictionary. The writer previously excluded any nullable column from global-dict candidacy, so a nullable low-cardinality string/numeric column (extremely common in real data — most Raincloud categorical columns have nulls) re-wrote its dictionary in every chunk. Nullable columns now run the same cardinality/ratio gate against their non-null values and emit one shared `vortex.dict` layout with per-chunk masked codes; the reader combines codes-side and pool-side validity per row. On a 200k-row, 10-chunk nullable low-cardinality Utf8 column this shrinks the file 54% (232 KB → 107 KB) and closes the gap to the Rust reference from ~4× to ~1.5×. (internal)
 - `MaskedEncodingEncoder` now encodes an all-valid or all-invalid validity bitmap as `vortex.constant` instead of a raw per-row bitmap. A nullable column chunk with zero nulls previously still paid a full `n/8`-byte bitmap for no information; an all-valid or all-invalid chunk now costs a few bytes regardless of row count. (internal)
 - `MaskedEncodingEncoder` also tries `vortex.sparse` for a mixed-validity bitmap (fill = the majority value, patches = the minority positions) and keeps it over the raw bitmap when smaller — the patch-index array is compressed through a dedicated `vortex.sequence`/`fastlanes.delta`/FoR/bitpack cascade, so a clustered or regular null pattern (e.g. one null every 10 rows) collapses to a handful of bytes instead of a full bitmap. On the 200k-row, 10-chunk nullable low-cardinality Utf8 benchmark this drops the file a further 20% (107 KB → 86 KB) and the gap to the Rust reference from ~1.45× to ~1.17×. Also: `SequenceEncodingEncoder.encodeCascade` no longer lets `encode`'s "not an arithmetic sequence" exception escape when a stratified sample looked arithmetic but the full data wasn't — it now reports the cascade step as not applicable, like every other encoder's contract requires. (internal)
+- `ConstantEncodingEncoder` now accepts `DType.Bool` (previously `DType.Primitive` only), so any all-true or all-false dense Bool column — not just a masked/nullable one — can win `vortex.constant` through the normal per-column cascade, not only through `MaskedEncodingEncoder`'s dedicated check. `MaskedEncodingEncoder`'s validity encoding now delegates to it instead of duplicating the scalar-building logic. `RunEndEncodingEncoder` also gains a `vortex.runend` path for Bool: `MaskedEncodingEncoder` tries it alongside `vortex.sparse` and keeps whichever is smallest — `vortex.runend` wins on long clustered runs of valid/invalid rows (regardless of which value dominates), a shape `vortex.sparse`'s per-flip patch cost handles poorly. (internal)
 
 ## [0.12.2] — 2026-07-12
 
diff --git a/TODO.md b/TODO.md
@@ -12,19 +12,15 @@
 - [ ] Performance tests must be peer-reviewed
 - [ ] Run performance tests on other machines (I have access only to Apple M5)
 - [ ] **Vector API adoption** — see [ADR-0005](adr/0005-vector-api-adoption.md).
-- [ ] **`DType.Bool` still has no RLE/Constant-at-the-`accepts()`-level encoding** —
-  `RunEndEncodingEncoder`/`RleEncodingEncoder`/`ConstantEncodingEncoder` all `accepts()` only
-  `DType.Primitive`. `MaskedEncodingEncoder` now covers the all-valid/all-invalid case directly
-  (`vortex.constant`, bypassing `accepts()`) and the mixed-validity case via a dedicated
-  `SparseEncodingEncoder.encodeBool` path (fill = majority value, patches = minority positions,
-  index array cascaded through `vortex.sequence`/`fastlanes.delta`/FoR/bitpack) — closing most of
-  the remaining vortex-jni gap on a clustered/regular null pattern (200k-row/10-chunk nullable
-  low-cardinality Utf8 benchmark: 1.45× → 1.17×, commits 5fe8b544/ecd47ead + Sparse-for-Bool).
-  What's left: a **scattered, non-clustered** null pattern (no exploitable index structure) still
-  costs close to the raw 1-bit/row bitmap — `vortex.runend`/`fastlanes.rle` genuinely support Bool
-  would help there, but that needs the same `accepts()`/fill-value generalization done for
-  `SparseEncodingEncoder`, applied to two more encoders (plus their decoders, if not already
-  Bool-capable like `SparseEncodingDecoder` turned out to be).
+- [ ] **`fastlanes.rle` still has no `DType.Bool` support** — `ConstantEncodingEncoder`,
+  `SparseEncodingEncoder`, and `RunEndEncodingEncoder` all now handle Bool (constant / scattered
+  minority / clustered runs respectively — `MaskedEncodingEncoder` tries all three plus a raw
+  bitmap and keeps the smallest), closing most of the vortex-jni gap on the 200k-row/10-chunk
+  nullable low-cardinality Utf8 benchmark (4× → 1.17×, commits 5fe8b544/ecd47ead/506d036f + this
+  session's RunEnd/Constant additions). `RleEncodingEncoder` (`fastlanes.rle`) is the one
+  remaining Primitive-only holdout; likely low marginal value now that Sparse and RunEnd cover
+  the scattered/clustered cases, but untested. A truly random (high-entropy) validity pattern has
+  no exploitable structure for any of these — that's an information-theoretic floor, not a gap.
 
 ## Security
 
diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/ConstantEncodingEncoder.java b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/ConstantEncodingEncoder.java
@@ -18,7 +18,7 @@ public EncodingId encodingId() {
 
     @Override
     public boolean accepts(DType dtype) {
-        return dtype instanceof DType.Primitive;
+        return dtype instanceof DType.Primitive || dtype instanceof DType.Bool;
     }
 
     @Override
@@ -42,8 +42,11 @@ public Estimate expectedRatio(DType dtype, Object data, ArrayStats stats) {
 
     @Override
     public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
+        if (dtype instanceof DType.Bool) {
+            return encodeBool((boolean[]) data);
+        }
         if (!(dtype instanceof DType.Primitive p)) {
-            throw new VortexException(EncodingId.VORTEX_CONSTANT, "encode only supports Primitive dtype, got " + dtype);
+            throw new VortexException(EncodingId.VORTEX_CONSTANT, "encode only supports Primitive or Bool dtype, got " + dtype);
         }
         PType ptype = p.ptype();
         if (!isConstant(data, ptype)) {
@@ -56,12 +59,40 @@ public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
 
     @Override
     public CascadeStep encodeCascade(DType dtype, Object data, EncodeContext encodeCtx) {
+        if (dtype instanceof DType.Bool bool) {
+            if (!isConstantBool((boolean[]) data)) {
+                return CascadeStep.notApplicable();
+            }
+            return CascadeStep.terminal(encode(bool, data, encodeCtx));
+        }
         if (!isConstant(data, ((DType.Primitive) dtype).ptype())) {
             return CascadeStep.notApplicable();
         }
         return CascadeStep.terminal(encode(dtype, data, encodeCtx));
     }
 
+    private static EncodeResult encodeBool(boolean[] data) {
+        if (!isConstantBool(data)) {
+            throw new VortexException(EncodingId.VORTEX_CONSTANT, "not a constant array");
+        }
+        boolean value = data.length == 0 || data[0];
+        ProtoScalarValue scalar = ProtoScalarValue.ofBoolValue(value);
+        return EncodeResult.simple(EncodingId.VORTEX_CONSTANT, MemorySegment.ofArray(scalar.encode()));
+    }
+
+    private static boolean isConstantBool(boolean[] data) {
+        if (data.length == 0) {
+            return true;
+        }
+        boolean first = data[0];
+        for (boolean b : data) {
+            if (b != first) {
+                return false;
+            }
+        }
+        return true;
+    }
+
     private static long readFirstRaw(Object data, PType ptype) {
         return switch (ptype) {
             case I8, U8 -> ((byte[]) data).length > 0 ? ((byte[]) data)[0] : 0L;
diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoder.java b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoder.java
@@ -4,7 +4,6 @@
 import io.github.dfa1.vortex.core.error.VortexException;
 
 import io.github.dfa1.vortex.core.model.EncodingId;
-import io.github.dfa1.vortex.core.proto.ProtoScalarValue;
 
 import java.lang.foreign.MemorySegment;
 import java.util.ArrayList;
@@ -86,27 +85,34 @@ private static EncodeResult encodeValues(DType nonNullable, Object values, Encod
     /// Encodes a masked column's validity bitmap.
     ///
     /// Every row sharing the same validity (the whole chunk all-valid or all-invalid) is
-    /// `vortex.constant` — a common case that otherwise costs a full `n/8`-byte bitmap for no
-    /// information. Otherwise, with cascade depth available, also tries `vortex.sparse`
-    /// ([SparseEncodingEncoder#encodeBool]) — a clustered or regular null pattern can compress its
-    /// patch-index array well below a raw bitmap — and keeps whichever candidate is smaller.
-    /// Without cascade depth, or when sparse doesn't win, falls back to a raw `vortex.bool` bitmap.
+    /// `vortex.constant` ([ConstantEncodingEncoder]) — a common case that otherwise costs a full
+    /// `n/8`-byte bitmap for no information. Otherwise, with cascade depth available, also tries
+    /// `vortex.sparse` ([SparseEncodingEncoder#encodeBool] — wins on a dominant value with
+    /// scattered rare flips, its patch-index array compressed further) and `vortex.runend`
+    /// ([RunEndEncodingEncoder#encodeBool] — wins on long clustered runs of valid/invalid rows,
+    /// regardless of which value dominates), keeping whichever candidate is smallest. Without
+    /// cascade depth, or when neither wins, falls back to a raw `vortex.bool` bitmap.
     ///
     /// @param validity per-row validity bitmap
     /// @param ctx      the encode context
     /// @return the encoded validity child
     private static EncodeResult encodeValidity(boolean[] validity, EncodeContext ctx) {
         if (isConstantValidity(validity)) {
-            boolean value = validity.length == 0 || validity[0];
-            ProtoScalarValue scalar = ProtoScalarValue.ofBoolValue(value);
-            return EncodeResult.simple(EncodingId.VORTEX_CONSTANT, MemorySegment.ofArray(scalar.encode()));
+            return new ConstantEncodingEncoder().encode(DType.BOOL, validity, ctx);
         }
-        EncodeResult raw = new BoolEncodingEncoder().encode(DType.BOOL, validity, ctx);
+        EncodeResult best = new BoolEncodingEncoder().encode(DType.BOOL, validity, ctx);
         if (ctx.allowedCascading() <= 0) {
-            return raw;
+            return best;
         }
         EncodeResult sparse = SparseEncodingEncoder.encodeBool(validity, ctx);
-        return totalBytes(sparse) < totalBytes(raw) ? sparse : raw;
+        if (totalBytes(sparse) < totalBytes(best)) {
+            best = sparse;
+        }
+        EncodeResult runEnd = RunEndEncodingEncoder.encodeBool(validity, ctx);
+        if (totalBytes(runEnd) < totalBytes(best)) {
+            best = runEnd;
+        }
+        return best;
     }
 
     private static long totalBytes(EncodeResult result) {
diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/RunEndEncodingEncoder.java b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/RunEndEncodingEncoder.java
@@ -49,6 +49,59 @@ public Estimate expectedRatio(DType dtype, Object data, ArrayStats stats) {
         return Estimate.COMPLETE;
     }
 
+    /// Encodes a boolean array as `vortex.runend`: consecutive equal values collapse into one run.
+    /// Unlike [SparseEncodingEncoder#encodeBool] (good for a dominant value with scattered rare
+    /// flips, at the cost of a patch per flip) this wins on CLUSTERED validity — long consecutive
+    /// stretches of valid or invalid rows — where every flip costs a run regardless of which value
+    /// is more frequent. Always builds the result; callers compare against alternatives (raw
+    /// bitmap, sparse) and keep whichever is smallest.
+    ///
+    /// @param validity per-row boolean array; must contain at least one `true` and one `false`
+    ///                 (an all-same array is cheaper as `vortex.constant`)
+    /// @param ctx      encode context
+    /// @return the encoded `vortex.runend` result
+    static EncodeResult encodeBool(boolean[] validity, EncodeContext ctx) {
+        int n = validity.length;
+        List<Integer> ends = new ArrayList<>();
+        List<Boolean> values = new ArrayList<>();
+        boolean runVal = validity[0];
+        for (int i = 1; i < n; i++) {
+            if (validity[i] != runVal) {
+                ends.add(i);
+                values.add(runVal);
+                runVal = validity[i];
+            }
+        }
+        ends.add(n);
+        values.add(runVal);
+
+        int numRuns = ends.size();
+        MemorySegment endsBuf = ctx.arena().allocate((long) numRuns * 4, 4);
+        for (int i = 0; i < numRuns; i++) {
+            endsBuf.setAtIndex(VortexFormat.LE_INT, i, ends.get(i));
+        }
+        boolean[] valuesArr = new boolean[numRuns];
+        for (int i = 0; i < numRuns; i++) {
+            valuesArr[i] = values.get(i);
+        }
+        EncodeResult valuesResult = new BoolEncodingEncoder().encode(DType.BOOL, valuesArr, ctx);
+
+        byte[] metaBytes = new ProtoRunEndMetadata(
+                io.github.dfa1.vortex.core.proto.ProtoPType.fromValue(PType.U32.ordinal()),
+                numRuns, 0L).encode();
+
+        EncodeNode endsNode = EncodeNode.leaf(EncodingId.VORTEX_PRIMITIVE, 0);
+        EncodeNode valuesNode = EncodeNode.remapBufferIndices(valuesResult.rootNode(), 1);
+
+        List<MemorySegment> buffers = new ArrayList<>();
+        buffers.add(endsBuf);
+        buffers.addAll(valuesResult.buffers());
+
+        EncodeNode root = new EncodeNode(EncodingId.VORTEX_RUNEND, MemorySegment.ofArray(metaBytes),
+                new EncodeNode[]{endsNode, valuesNode}, new int[0]);
+        return new EncodeResult(root, List.copyOf(buffers), null, null);
+    }
+
     @Override
     public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
         if (!(dtype instanceof DType.Primitive p)) {
diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/encode/ConstantEncodingEncoderTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/encode/ConstantEncodingEncoderTest.java
@@ -2,13 +2,15 @@
 
 import io.github.dfa1.vortex.core.model.DType;
 import io.github.dfa1.vortex.core.model.EncodingId;
+import io.github.dfa1.vortex.core.error.VortexException;
 import io.github.dfa1.vortex.core.proto.ProtoNullValue;
 import io.github.dfa1.vortex.core.proto.ProtoScalarValue;
 import io.github.dfa1.vortex.reader.array.Array;
 import io.github.dfa1.vortex.reader.array.ByteArray;
 import io.github.dfa1.vortex.reader.array.DoubleArray;
 import io.github.dfa1.vortex.reader.array.FloatArray;
 import io.github.dfa1.vortex.reader.array.IntArray;
+import io.github.dfa1.vortex.reader.array.LazyConstantBoolArray;
 import io.github.dfa1.vortex.reader.array.LazyConstantIntArray;
 import io.github.dfa1.vortex.reader.array.LazyConstantLongArray;
 import io.github.dfa1.vortex.reader.array.LongArray;
@@ -31,6 +33,7 @@
 import java.util.stream.Stream;
 
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
 
 /// Property: encode then decode is lossless for constant (all-equal) arrays.
 /// Property: decode emits a metadata-only `LazyConstantXxxArray` — no buffer at any rowCount.
@@ -245,6 +248,50 @@ void i8_getByte_broadcastsAcrossEveryIndex() {
         }
     }
 
+    @Nested
+    class Bool {
+
+        @ParameterizedTest
+        @ValueSource(booleans = {true, false})
+        void encodeDecode_isLossless(boolean value) {
+            // Given / When
+            boolean[] data = {value, value, value};
+            EncodeResult resultEncoded = ENCODER.encode(DType.BOOL, data, EncodeTestHelper.testCtx());
+            DecodeContext ctx = DecodeTestHelper.toDecodeContext(resultEncoded, data.length, DType.BOOL, REGISTRY);
+            Array result = DECODER.decode(ctx);
+
+            // Then
+            assertThat(result.length()).isEqualTo(data.length);
+            assertThat(result).isInstanceOf(LazyConstantBoolArray.class);
+            assertThat(((LazyConstantBoolArray) result).value()).isEqualTo(value);
+        }
+
+        @Test
+        void encode_mixedValues_throws() {
+            // Given
+            boolean[] data = {true, false};
+
+            // When
+            // Then
+            assertThatThrownBy(() -> ENCODER.encode(DType.BOOL, data, EncodeTestHelper.testCtx()))
+                    .isInstanceOf(VortexException.class)
+                    .hasMessageContaining("not a constant array");
+        }
+
+        @Test
+        void encodeCascade_mixedValues_notApplicable() {
+            // Given — the cascade contract needs a non-applicable step, not an exception, so a
+            // sample-selected winner that isn't constant on the full data can be excluded and retried
+            boolean[] data = {true, false};
+
+            // When
+            CascadeStep step = ENCODER.encodeCascade(DType.BOOL, data, EncodeTestHelper.testCtx());
+
+            // Then
+            assertThat(step.applicable()).isFalse();
+        }
+    }
+
     /// Rust can write a constant array whose scalar is null (proto null_value tag).
     /// The decoder must return a [NullArray] — not 0 / false (#246).
     @Nested
diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoderTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoderTest.java
@@ -300,4 +300,34 @@ void withCascade_denseRandomNulls_keepsRawBitmap() {
             assertThat(masked.isValid(i)).as("row %d", i).isEqualTo(validity[i]);
         }
     }
+
+    @Test
+    void withCascade_clusteredNulls_prefersRunEndOverSparseAndRawBitmap() {
+        // Given — 2000 rows, one contiguous invalid stretch (rows 500-999): a handful of runs,
+        // where sparse would pay a patch per invalid row but run-end pays only for the two
+        // boundaries — the shape run-end targets that sparse does not.
+        int n = 2_000;
+        int[] values = new int[n];
+        boolean[] validity = new boolean[n];
+        for (int i = 0; i < n; i++) {
+            values[i] = i;
+            validity[i] = i < 500 || i >= 1_000;
+        }
+        DType i32Nullable = new DType.Primitive(PType.I32, true);
+        NullableData data = new NullableData(values, validity);
+        EncodeContext cascadeCtx = EncodeContext.ofDepth(3, Arena.ofAuto(), WriteRegistry.loadAll());
+
+        // When
+        EncodeResult result = SUT.encode(i32Nullable, data, cascadeCtx);
+
+        // Then — the validity child picked vortex.runend over sparse and a raw bitmap
+        assertThat(result.rootNode().children()[1].encodingId()).isEqualTo(EncodingId.VORTEX_RUNEND);
+
+        // And every value and null position round-trips exactly
+        Array decoded = DECODER.decode(DecodeTestHelper.toDecodeContext(result, (long) n, i32Nullable, ReadRegistry.loadAll()));
+        MaskedArray masked = (MaskedArray) decoded;
+        for (int i = 0; i < n; i++) {
+            assertThat(masked.isValid(i)).as("row %d", i).isEqualTo(validity[i]);
+        }
+    }
 }
