perf: fastlanes.rle for Bool, closing encoding parity

dfa1 · claude · dfa1 · commit fe6f132b7a7b · 2026-07-12T18:58:27.000+02:00
RleEncodingEncoder/RleEncodingDecoder gain Bool support: the same
FastLanes 1024-row chunked layout as the numeric path, with a
vortex.bool-encoded values pool (at most 2 distinct values per
chunk) instead of a ptype-width one. Unlike vortex.sparse/
vortex.runend (reader was already Bool-capable), fastlanes.rle
needed both sides: a new LazyRleBoolArray plus a decode branch,
since RleEncodingDecoder previously rejected any non-Primitive
dtype outright.

MaskedEncodingEncoder now tries all four validity encodings
(vortex.constant, vortex.sparse, vortex.runend, fastlanes.rle)
and keeps whichever is smallest. DType.Bool now has every
general-purpose encoding family DType.Primitive has.

Drops the now-closed TODO.md gap.

Co-Authored-By: Claude Sonnet 5 &lt;noreply@anthropic.com&gt;
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `MaskedEncodingEncoder` now encodes an all-valid or all-invalid validity bitmap as `vortex.constant` instead of a raw per-row bitmap. A nullable column chunk with zero nulls previously still paid a full `n/8`-byte bitmap for no information; an all-valid or all-invalid chunk now costs a few bytes regardless of row count. (internal)
 - `MaskedEncodingEncoder` also tries `vortex.sparse` for a mixed-validity bitmap (fill = the majority value, patches = the minority positions) and keeps it over the raw bitmap when smaller — the patch-index array is compressed through a dedicated `vortex.sequence`/`fastlanes.delta`/FoR/bitpack cascade, so a clustered or regular null pattern (e.g. one null every 10 rows) collapses to a handful of bytes instead of a full bitmap. On the 200k-row, 10-chunk nullable low-cardinality Utf8 benchmark this drops the file a further 20% (107 KB → 86 KB) and the gap to the Rust reference from ~1.45× to ~1.17×. Also: `SequenceEncodingEncoder.encodeCascade` no longer lets `encode`'s "not an arithmetic sequence" exception escape when a stratified sample looked arithmetic but the full data wasn't — it now reports the cascade step as not applicable, like every other encoder's contract requires. (internal)
 - `ConstantEncodingEncoder` now accepts `DType.Bool` (previously `DType.Primitive` only), so any all-true or all-false dense Bool column — not just a masked/nullable one — can win `vortex.constant` through the normal per-column cascade, not only through `MaskedEncodingEncoder`'s dedicated check. `MaskedEncodingEncoder`'s validity encoding now delegates to it instead of duplicating the scalar-building logic. `RunEndEncodingEncoder` also gains a `vortex.runend` path for Bool: `MaskedEncodingEncoder` tries it alongside `vortex.sparse` and keeps whichever is smallest — `vortex.runend` wins on long clustered runs of valid/invalid rows (regardless of which value dominates), a shape `vortex.sparse`'s per-flip patch cost handles poorly. (internal)
+- `RleEncodingEncoder`/`RleEncodingDecoder` (`fastlanes.rle`) round out Bool coverage: the same FastLanes 1024-row chunked layout the numeric path uses, with a `vortex.bool`-encoded values pool (at most 2 distinct values per chunk) instead of a ptype-width one. `MaskedEncodingEncoder` tries it alongside `vortex.sparse`/`vortex.runend` and keeps the smallest — `DType.Bool` now has every general-purpose encoding family (`vortex.constant`, `vortex.sparse`, `vortex.runend`, `fastlanes.rle`) that `DType.Primitive` has. (internal)
 
 ## [0.12.2] — 2026-07-12
 
diff --git a/TODO.md b/TODO.md
@@ -12,15 +12,6 @@
 - [ ] Performance tests must be peer-reviewed
 - [ ] Run performance tests on other machines (I have access only to Apple M5)
 - [ ] **Vector API adoption** — see [ADR-0005](adr/0005-vector-api-adoption.md).
-- [ ] **`fastlanes.rle` still has no `DType.Bool` support** — `ConstantEncodingEncoder`,
-  `SparseEncodingEncoder`, and `RunEndEncodingEncoder` all now handle Bool (constant / scattered
-  minority / clustered runs respectively — `MaskedEncodingEncoder` tries all three plus a raw
-  bitmap and keeps the smallest), closing most of the vortex-jni gap on the 200k-row/10-chunk
-  nullable low-cardinality Utf8 benchmark (4× → 1.17×, commits 5fe8b544/ecd47ead/506d036f + this
-  session's RunEnd/Constant additions). `RleEncodingEncoder` (`fastlanes.rle`) is the one
-  remaining Primitive-only holdout; likely low marginal value now that Sparse and RunEnd cover
-  the scattered/clustered cases, but untested. A truly random (high-entropy) validity pattern has
-  no exploitable structure for any of these — that's an information-theoretic floor, not a gap.
 
 ## Security
 
diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/array/LazyRleBoolArray.java b/reader/src/main/java/io/github/dfa1/vortex/reader/array/LazyRleBoolArray.java
@@ -0,0 +1,69 @@
+package io.github.dfa1.vortex.reader.array;
+
+import io.github.dfa1.vortex.core.model.DType;
+
+/// Lazy FastLanes-RLE-encoded [BoolArray].
+///
+/// Mirrors [LazyRleLongArray] — see its doc for the chunk/values/indices layout — with `values`
+/// a [BoolArray] (at most 2 distinct values per 1024-row chunk) instead of a numeric array.
+///
+/// @param dtype             logical Bool type
+/// @param length            total logical row count
+/// @param values            concatenated distinct values per chunk
+/// @param indices           per-row local index table; length `numChunks * 1024`
+/// @param valuesIdxOffsets  per-chunk values-pool start offsets; length `numChunks`
+/// @param firstOffset       absolute origin of the values pool
+/// @param valuesLen         total values pool length
+/// @param numChunks         number of FastLanes chunks covered
+/// @param offset            starting absolute position; logical row `i` maps to
+///                          absolute `i + offset`
+@SuppressWarnings("java:S6218") // internal data carrier; record components are arrays of immutable primitives or refs that flow through pipelines without ever being compared.
+public record LazyRleBoolArray(
+        DType dtype, long length, BoolArray values, int[] indices,
+        long[] valuesIdxOffsets, long firstOffset, long valuesLen,
+        int numChunks, int offset)
+        implements BoolArray {
+
+    @Override
+    public boolean getBoolean(long i) {
+        int absRow = (int) (i + offset);
+        int chunkIdx = absRow >>> RleArrays.FL_LOG2;
+        int rowInChunk = absRow & RleArrays.FL_MASK;
+        long valueIdxOffset = valuesIdxOffsets[chunkIdx] - firstOffset;
+        int numChunkValues = RleArrays.chunkValueCount(chunkIdx, numChunks, valuesIdxOffsets, firstOffset, valuesLen);
+        if (numChunkValues <= 1) {
+            return numChunkValues == 1 && values.getBoolean(valueIdxOffset);
+        }
+        int localIdx = indices[chunkIdx * RleArrays.FL_CHUNK_SIZE + rowInChunk];
+        if (localIdx >= numChunkValues) {
+            localIdx = numChunkValues - 1;
+        }
+        return values.getBoolean(valueIdxOffset + localIdx);
+    }
+
+    @Override
+    public void forEachBoolean(BooleanConsumer c) {
+        RleArrays.walkChunks(length, offset, numChunks,
+                (chunkIdx, rowInChunk, end) -> processChunk(chunkIdx, rowInChunk, end, c));
+    }
+
+    private void processChunk(int chunkIdx, int rowInChunk, int end, BooleanConsumer c) {
+        int chunkBase = chunkIdx * RleArrays.FL_CHUNK_SIZE;
+        long valueIdxOffset = valuesIdxOffsets[chunkIdx] - firstOffset;
+        int numChunkValues = RleArrays.chunkValueCount(chunkIdx, numChunks, valuesIdxOffsets, firstOffset, valuesLen);
+        if (numChunkValues <= 1) {
+            boolean v = numChunkValues == 1 && values.getBoolean(valueIdxOffset);
+            for (int r = rowInChunk; r < end; r++) {
+                c.accept(v);
+            }
+        } else {
+            for (int r = rowInChunk; r < end; r++) {
+                int localIdx = indices[chunkBase + r];
+                if (localIdx >= numChunkValues) {
+                    localIdx = numChunkValues - 1;
+                }
+                c.accept(values.getBoolean(valueIdxOffset + localIdx));
+            }
+        }
+    }
+}
diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/decode/RleEncodingDecoder.java b/reader/src/main/java/io/github/dfa1/vortex/reader/decode/RleEncodingDecoder.java
@@ -14,6 +14,7 @@
 import io.github.dfa1.vortex.reader.array.LazyConstantIntArray;
 import io.github.dfa1.vortex.reader.array.LazyConstantLongArray;
 import io.github.dfa1.vortex.reader.array.LazyConstantShortArray;
+import io.github.dfa1.vortex.reader.array.LazyRleBoolArray;
 import io.github.dfa1.vortex.reader.array.LazyRleByteArray;
 import io.github.dfa1.vortex.reader.array.LazyRleDoubleArray;
 import io.github.dfa1.vortex.reader.array.LazyRleFloatArray;
@@ -59,12 +60,6 @@ public Array decode(DecodeContext ctx) {
             return emptyArray(ctx);
         }
 
-        if (!(ctx.dtype() instanceof DType.Primitive p)) {
-            throw new VortexException(EncodingId.FASTLANES_RLE, "expected Primitive dtype, got " + ctx.dtype());
-        }
-        PType ptype = p.ptype();
-
-        DType valuesDtype = new DType.Primitive(ptype, false);
         DType indicesDtype = new DType.Primitive(indicesPtype, false);
         DType offsetsDtype = new DType.Primitive(offsetsPtype, false);
 
@@ -83,6 +78,23 @@ public Array decode(DecodeContext ctx) {
         long firstOffset = valuesLen > 0 && valuesIdxOffsets.length > 0 ? valuesIdxOffsets[0] : 0L;
         int numChunks = (int) (indicesLen / FL_CHUNK_SIZE);
 
+        if (ctx.dtype() instanceof DType.Bool) {
+            Array valuesArr = ctx.decodeChild(0, DType.BOOL, valuesLen);
+            Array valuesData = valuesArr instanceof MaskedArray m ? m.inner() : valuesArr;
+            Array boolResult = new LazyRleBoolArray(ctx.dtype(), rowCount, (BoolArray) valuesData,
+                    indices, valuesIdxOffsets, firstOffset, valuesLen, numChunks, offset);
+            if (indicesValidity == null) {
+                return boolResult;
+            }
+            return new MaskedArray(boolResult, new OffsetBoolArray(DType.BOOL, rowCount, indicesValidity, offset));
+        }
+
+        if (!(ctx.dtype() instanceof DType.Primitive p)) {
+            throw new VortexException(EncodingId.FASTLANES_RLE, "expected Primitive dtype, got " + ctx.dtype());
+        }
+        PType ptype = p.ptype();
+        DType valuesDtype = new DType.Primitive(ptype, false);
+
         MemorySegment valuesSeg = ctx.decodeChildSegment(0, valuesDtype, valuesLen);
         Array result = switch (ptype) {
             case I64, U64 -> new LazyRleLongArray(ctx.dtype(), rowCount,
diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoder.java b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/MaskedEncodingEncoder.java
@@ -88,10 +88,12 @@ private static EncodeResult encodeValues(DType nonNullable, Object values, Encod
     /// `vortex.constant` ([ConstantEncodingEncoder]) — a common case that otherwise costs a full
     /// `n/8`-byte bitmap for no information. Otherwise, with cascade depth available, also tries
     /// `vortex.sparse` ([SparseEncodingEncoder#encodeBool] — wins on a dominant value with
-    /// scattered rare flips, its patch-index array compressed further) and `vortex.runend`
+    /// scattered rare flips, its patch-index array compressed further), `vortex.runend`
     /// ([RunEndEncodingEncoder#encodeBool] — wins on long clustered runs of valid/invalid rows,
-    /// regardless of which value dominates), keeping whichever candidate is smallest. Without
-    /// cascade depth, or when neither wins, falls back to a raw `vortex.bool` bitmap.
+    /// regardless of which value dominates), and `fastlanes.rle`
+    /// ([RleEncodingEncoder#encodeBool] — the same clustered-run shape as run-end, FastLanes'
+    /// vectorized chunked layout instead), keeping whichever candidate is smallest. Without
+    /// cascade depth, or when none wins, falls back to a raw `vortex.bool` bitmap.
     ///
     /// @param validity per-row validity bitmap
     /// @param ctx      the encode context
@@ -112,6 +114,10 @@ private static EncodeResult encodeValidity(boolean[] validity, EncodeContext ctx
         if (totalBytes(runEnd) < totalBytes(best)) {
             best = runEnd;
         }
+        EncodeResult rle = RleEncodingEncoder.encodeBool(validity, ctx);
+        if (totalBytes(rle) < totalBytes(best)) {
+            best = rle;
+        }
         return best;
     }
 
diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/RleEncodingEncoder.java b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/RleEncodingEncoder.java
@@ -11,6 +11,7 @@
 
 import java.lang.foreign.MemorySegment;
 import java.lang.foreign.SegmentAllocator;
+import java.util.ArrayList;
 import java.util.List;
 
 /// Write-only encoder for `fastlanes.rle`.
@@ -28,6 +29,90 @@ public boolean accepts(DType dtype) {
         return dtype instanceof DType.Primitive p && !p.ptype().isFloating();
     }
 
+    /// Encodes a boolean array as `fastlanes.rle`: the same FastLanes 1024-row chunked
+    /// layout as the numeric path (see [#encode]), with a `vortex.bool`-encoded values pool
+    /// (at most 2 distinct values per chunk for a two-valued domain) instead of a ptype-width
+    /// primitive one. Complements [RunEndEncodingEncoder#encodeBool] (plain run-ends); callers
+    /// compare against alternatives and keep whichever is smallest.
+    ///
+    /// @param validity per-row boolean array; must contain at least one `true` and one `false`
+    /// @param ctx      encode context
+    /// @return the encoded `fastlanes.rle` result
+    static EncodeResult encodeBool(boolean[] validity, EncodeContext ctx) {
+        int n = validity.length;
+        long[] longs = new long[n];
+        for (int i = 0; i < n; i++) {
+            longs[i] = validity[i] ? 1L : 0L;
+        }
+
+        int numChunks = (n + FL_CHUNK_SIZE - 1) / FL_CHUNK_SIZE;
+        int paddedLen = numChunks * FL_CHUNK_SIZE;
+
+        long[] globalValues = new long[paddedLen];
+        short[] globalIndices = new short[paddedLen];
+        long[] valuesIdxOffsets = new long[numChunks];
+
+        long[] chunkInput = new long[FL_CHUNK_SIZE];
+        long[] chunkValues = new long[FL_CHUNK_SIZE];
+        short[] chunkIndices = new short[FL_CHUNK_SIZE];
+
+        int globalValuesCount = 0;
+        for (int chunk = 0; chunk < numChunks; chunk++) {
+            int chunkStart = chunk * FL_CHUNK_SIZE;
+            int chunkEnd = Math.min(chunkStart + FL_CHUNK_SIZE, n);
+            int chunkLen = chunkEnd - chunkStart;
+
+            System.arraycopy(longs, chunkStart, chunkInput, 0, chunkLen);
+            long lastVal = longs[chunkEnd - 1];
+            for (int i = chunkLen; i < FL_CHUNK_SIZE; i++) {
+                chunkInput[i] = lastVal;
+            }
+
+            int numChunkValues = rleEncode(chunkInput, chunkValues, chunkIndices);
+
+            valuesIdxOffsets[chunk] = globalValuesCount;
+            System.arraycopy(chunkValues, 0, globalValues, globalValuesCount, numChunkValues);
+            globalValuesCount += numChunkValues;
+
+            System.arraycopy(chunkIndices, 0, globalIndices, chunkStart, FL_CHUNK_SIZE);
+        }
+
+        boolean[] valuesArr = new boolean[globalValuesCount];
+        for (int i = 0; i < globalValuesCount; i++) {
+            valuesArr[i] = globalValues[i] != 0L;
+        }
+        EncodeResult valuesResult = new BoolEncodingEncoder().encode(DType.BOOL, valuesArr, ctx);
+        MemorySegment indicesSeg = toIndicesSeg(globalIndices, paddedLen, ctx.arena());
+        MemorySegment offsetsSeg = fromLongsU64(valuesIdxOffsets, numChunks, ctx.arena());
+
+        PType indicesPtype = PType.U16;
+        PType offsetsPtype = PType.U64;
+
+        byte[] metaBytes = new ProtoRLEMetadata(
+                globalValuesCount,
+                paddedLen,
+                io.github.dfa1.vortex.core.proto.ProtoPType.fromValue(indicesPtype.ordinal()),
+                numChunks,
+                io.github.dfa1.vortex.core.proto.ProtoPType.fromValue(offsetsPtype.ordinal()),
+                0L
+        ).encode();
+
+        int indicesBufIdx = valuesResult.buffers().size();
+        EncodeNode indicesNode = EncodeNode.leaf(EncodingId.VORTEX_PRIMITIVE, indicesBufIdx);
+        EncodeNode offsetsNode = EncodeNode.leaf(EncodingId.VORTEX_PRIMITIVE, indicesBufIdx + 1);
+
+        List<MemorySegment> buffers = new ArrayList<>(valuesResult.buffers());
+        buffers.add(indicesSeg);
+        buffers.add(offsetsSeg);
+
+        EncodeNode root = new EncodeNode(
+                EncodingId.FASTLANES_RLE,
+                MemorySegment.ofArray(metaBytes),
+                new EncodeNode[]{valuesResult.rootNode(), indicesNode, offsetsNode},
+                new int[0]);
+        return new EncodeResult(root, List.copyOf(buffers), null, null);
+    }
+
     @Override
     public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
         if (!(dtype instanceof DType.Primitive p)) {
diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/encode/RleEncodingEncoderTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/encode/RleEncodingEncoderTest.java
@@ -213,6 +213,58 @@ void roundTrip_negativeValues_i32() {
         }
     }
 
+    @Nested
+    class Bool {
+
+        private static final ReadRegistry BOOL_REGISTRY = TestRegistry.ofDecoders(
+                DECODER, new PrimitiveEncodingDecoder(), new BoolEncodingDecoder());
+
+        @Test
+        void roundTrip_clusteredRuns() {
+            // Given — 2000 rows, one contiguous invalid stretch (rows 500-999), spanning multiple
+            // FastLanes 1024-row chunks.
+            int n = 2_000;
+            boolean[] data = new boolean[n];
+            for (int i = 0; i < n; i++) {
+                data[i] = i < 500 || i >= 1_000;
+            }
+
+            // When
+            EncodeResult encoded = RleEncodingEncoder.encodeBool(data, EncodeTestHelper.testCtx());
+            DecodeContext ctx = DecodeTestHelper.toDecodeContext(encoded, n, DType.BOOL, BOOL_REGISTRY);
+            Array result = DECODER.decode(ctx);
+
+            // Then
+            assertThat(result.length()).isEqualTo(n);
+            io.github.dfa1.vortex.reader.array.BoolArray bools = (io.github.dfa1.vortex.reader.array.BoolArray) result;
+            for (int i = 0; i < n; i++) {
+                assertThat(bools.getBoolean(i)).as("index %d", i).isEqualTo(data[i]);
+            }
+        }
+
+        @ParameterizedTest
+        @ValueSource(ints = {1, 512, 1023, 1024, 1025, 2048, 2049})
+        void roundTrip_variousLengths(int n) {
+            // Given — alternating short runs of true/false, various lengths around chunk boundaries
+            boolean[] data = new boolean[n];
+            for (int i = 0; i < n; i++) {
+                data[i] = (i / 7) % 2 == 0;
+            }
+
+            // When
+            EncodeResult encoded = RleEncodingEncoder.encodeBool(data, EncodeTestHelper.testCtx());
+            DecodeContext ctx = DecodeTestHelper.toDecodeContext(encoded, n, DType.BOOL, BOOL_REGISTRY);
+            Array result = DECODER.decode(ctx);
+
+            // Then
+            assertThat(result.length()).isEqualTo(n);
+            io.github.dfa1.vortex.reader.array.BoolArray bools = (io.github.dfa1.vortex.reader.array.BoolArray) result;
+            for (int i = 0; i < n; i++) {
+                assertThat(bools.getBoolean(i)).as("index %d", i).isEqualTo(data[i]);
+            }
+        }
+    }
+
     @Nested
     class Decode {
 
