fix: lazy dict decode for I8/U8/I16/U16 value columns (#206)

dfa1 · claude · dfa1 · commit a9e41373479b · 2026-07-06T21:21:18.000+02:00
Files from the Python bindings (vortex-data 0.69.0) dict-encode narrow-integer columns; the dict values land as U8/U16 primitives and buildLazyDictPrimitive threw "unsupported ptype for lazy dict". Add DictByteArray and DictShortArray mirroring the existing lazy dict records: raw-signed storage semantics (unsigned widening stays in the ByteArray/ShortArray getInt defaults, keyed on dtype), hoisted code-width switches in materialize/forEach/fold, and the same validateCodes construction guard. F16 remains the only unsupported value ptype. On the Raincloud corpus this flips uci-bike-sharing-dataset to a full value-for-value pass and unblocks the kepler/penguins scans far enough to expose two pre-existing bugs, reclassified in the conformance matrix: nullable ALP columns dropping row validity (#210) and CSV export lacking a NullArray arm (#211). Closes #206 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- Lazy dict decode now covers I8/U8/I16/U16 value columns (`DictByteArray`, `DictShortArray`) — files from the Python bindings dict-encode narrow-integer columns, which previously failed to scan with `unsupported ptype for lazy dict`. ([#206](https://github.com/dfa1/vortex-java/issues/206))
+
 ### Added
 
 - Real-world conformance suite against the [Raincloud](https://github.com/spiraldb/raincloud) corpus: 247 public datasets whose Vortex files are written by the Python bindings, each validated value-for-value against its Parquet sibling. `scripts/hydrate-raincloud-corpus.sh` hydrates any subset (cache → mirror → local build), `RaincloudConformanceIntegrationTest` tests whatever is hydrated against the checked-in per-dataset status matrix, and a weekly workflow runs a size-capped sweep. First triage found four reader gaps on real data, including one silent-corruption bug (unsigned integers rendered as signed). ([#205](https://github.com/dfa1/vortex-java/issues/205))
diff --git a/docs/compatibility.md b/docs/compatibility.md
@@ -62,11 +62,14 @@ Per-slug status lives in `integration/src/test/resources/raincloud/expected-stat
 (`ok` must pass; `gap:<issue>` must still fail, so a fix flips the entry in the same change;
 `untriaged` runs and reports without failing the build). A scheduled workflow
 (`raincloud-conformance.yml`) hydrates a size-capped subset weekly. Current triage —
-4 `ok`, 6 known gaps: lazy dict U8/U16 values
-([#206](https://github.com/dfa1/vortex-java/issues/206)), nested struct columns in scan
+5 `ok`, 5 known gaps: nested struct columns in scan
 ([#207](https://github.com/dfa1/vortex-java/issues/207)), unsigned integers rendered signed
 ([#208](https://github.com/dfa1/vortex-java/issues/208) — silent corruption), RLE over F64
-([#209](https://github.com/dfa1/vortex-java/issues/209)); 237 slugs untriaged.
+([#209](https://github.com/dfa1/vortex-java/issues/209)), nullable ALP validity dropped
+([#210](https://github.com/dfa1/vortex-java/issues/210) — silent corruption), all-null
+columns in CSV export ([#211](https://github.com/dfa1/vortex-java/issues/211));
+237 slugs untriaged. Fixed so far by this suite: lazy dict U8/U16 values
+([#206](https://github.com/dfa1/vortex-java/issues/206)).
 
 ## Encodings
 
diff --git a/integration/src/test/resources/raincloud/expected-status.csv b/integration/src/test/resources/raincloud/expected-status.csv
@@ -124,7 +124,7 @@ humaneval,untriaged
 insurance,untriaged
 international-football-results-from-1872-to-2017,untriaged
 iowa-liquor-sales,untriaged
-kepler-exoplanet-search-results,gap:206
+kepler-exoplanet-search-results,gap:211
 kepler-labelled-time-series-data,untriaged
 laion-400m,untriaged
 librispeech-test-clean,untriaged
@@ -166,7 +166,7 @@ osmi-mental-health-in-tech-2020,untriaged
 osmi-mental-health-in-tech-2021,untriaged
 osmi-mental-health-in-tech-2022,untriaged
 osmi-mental-health-in-tech-2023,untriaged
-palmer-archipelago-antarctica-penguin-data,gap:206
+palmer-archipelago-antarctica-penguin-data,gap:210
 peoples-speech-clean-validation,untriaged
 personal-key-indicators-of-heart-disease,untriaged
 pleias-synth,untriaged
@@ -194,7 +194,7 @@ uci-auto-mpg,untriaged
 uci-automobile,untriaged
 uci-bank-marketing,untriaged
 uci-beijing-multi-site-air-quality,untriaged
-uci-bike-sharing-dataset,gap:206
+uci-bike-sharing-dataset,ok
 uci-breast-cancer,untriaged
 uci-breast-cancer-wisconsin-diagnostic,untriaged
 uci-breast-cancer-wisconsin-original,untriaged
diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/array/DictByteArray.java b/reader/src/main/java/io/github/dfa1/vortex/reader/array/DictByteArray.java
@@ -0,0 +1,150 @@
+package io.github.dfa1.vortex.reader.array;
+
+import io.github.dfa1.vortex.core.model.DType;
+import io.github.dfa1.vortex.core.error.VortexException;
+
+import java.lang.foreign.MemorySegment;
+import java.lang.foreign.SegmentAllocator;
+import java.lang.foreign.ValueLayout;
+import java.util.function.LongBinaryOperator;
+
+/// Dict-encoded [ByteArray] view. ADR 0012 shape.
+///
+/// Stores `values` (the dictionary pool) and `codes` (one index per
+/// row into `values`). Scalar access resolves on demand:
+/// `getByte(i) = values.getByte(codes.getCode(i))`. Per ADR 0012, this
+/// preserves zero-copy on dict-encoded categorical columns.
+///
+/// The `codes` array is typed as [Array] because the codes ptype
+/// varies with dictionary size — U8/U16/U32/U64 backed by
+/// [ByteArray]/[ShortArray]/[IntArray]/[LongArray].
+///
+/// @param dtype  logical element type (matches `values.dtype()`)
+/// @param length total logical row count (matches `codes.length()`)
+/// @param values dictionary pool — element at code `c` is `values.getByte(c)`
+/// @param codes  per-row index into `values`; must be one of
+///               [ByteArray], [ShortArray], [IntArray], [LongArray]
+public record DictByteArray(DType dtype, long length, ByteArray values, Array codes) implements ByteArray {
+
+    /// Builds a [DictByteArray], validating that `codes` is one of the
+    /// four narrow-int code array types and that its length matches `length`.
+    ///
+    /// @param dtype  logical element type
+    /// @param length total logical row count
+    /// @param values dictionary pool
+    /// @param codes  per-row code array (must be [ByteArray], [ShortArray],
+    ///               [IntArray], or [LongArray])
+    /// @return a new [DictByteArray]
+    /// @throws VortexException if `codes` is not a supported code-array type or
+    ///                         its length does not equal `length`
+    public static DictByteArray of(DType dtype, long length, ByteArray values, Array codes) {
+        DictArrays.validateCodes(codes, length);
+        return new DictByteArray(dtype, length, values, codes);
+    }
+
+    @Override
+    public byte getByte(long i) {
+        return values.getByte(DictArrays.readCode(codes, i));
+    }
+
+    /// Materializes by gathering one dictionary value per code into a fresh
+    /// one-byte-per-element segment. The codes switch is hoisted outside the loop so
+    /// each branch is a uniform gather over a single code width.
+    ///
+    /// @param arena allocator for the output segment
+    /// @return a read-only segment of `length()` gathered bytes
+    /// @throws VortexException if `codes` is not a supported code-array type
+    @Override
+    public MemorySegment materialize(SegmentAllocator arena) {
+        long n = length;
+        MemorySegment dst = arena.allocate(n);
+        ByteArray vals = values;
+        switch (codes) {
+            case ByteArray ba -> {
+                for (long i = 0; i < n; i++) {
+                    dst.set(ValueLayout.JAVA_BYTE, i, vals.getByte(Byte.toUnsignedLong(ba.getByte(i))));
+                }
+            }
+            case ShortArray sa -> {
+                for (long i = 0; i < n; i++) {
+                    dst.set(ValueLayout.JAVA_BYTE, i, vals.getByte(Short.toUnsignedLong(sa.getShort(i))));
+                }
+            }
+            case IntArray ia -> {
+                for (long i = 0; i < n; i++) {
+                    dst.set(ValueLayout.JAVA_BYTE, i, vals.getByte(Integer.toUnsignedLong(ia.getInt(i))));
+                }
+            }
+            case LongArray la -> {
+                for (long i = 0; i < n; i++) {
+                    dst.set(ValueLayout.JAVA_BYTE, i, vals.getByte(la.getLong(i)));
+                }
+            }
+            default -> throw new VortexException("DictByteArray: invalid codes type: "
+                    + codes.getClass().getSimpleName());
+        }
+        return dst.asReadOnly();
+    }
+
+    @Override
+    public void forEachByte(ByteConsumer cons) {
+        long n = length;
+        ByteArray vals = values;
+        switch (codes) {
+            case ByteArray ba -> {
+                for (long i = 0; i < n; i++) {
+                    cons.accept(vals.getByte(Byte.toUnsignedLong(ba.getByte(i))));
+                }
+            }
+            case ShortArray sa -> {
+                for (long i = 0; i < n; i++) {
+                    cons.accept(vals.getByte(Short.toUnsignedLong(sa.getShort(i))));
+                }
+            }
+            case IntArray ia -> {
+                for (long i = 0; i < n; i++) {
+                    cons.accept(vals.getByte(Integer.toUnsignedLong(ia.getInt(i))));
+                }
+            }
+            case LongArray la -> {
+                for (long i = 0; i < n; i++) {
+                    cons.accept(vals.getByte(la.getLong(i)));
+                }
+            }
+            default -> throw new VortexException("DictByteArray: invalid codes type: "
+                    + codes.getClass().getSimpleName());
+        }
+    }
+
+    @Override
+    public long fold(long identity, LongBinaryOperator op) {
+        long n = length;
+        ByteArray vals = values;
+        long result = identity;
+        switch (codes) {
+            case ByteArray ba -> {
+                for (long i = 0; i < n; i++) {
+                    result = op.applyAsLong(result, vals.getByte(Byte.toUnsignedLong(ba.getByte(i))));
+                }
+            }
+            case ShortArray sa -> {
+                for (long i = 0; i < n; i++) {
+                    result = op.applyAsLong(result, vals.getByte(Short.toUnsignedLong(sa.getShort(i))));
+                }
+            }
+            case IntArray ia -> {
+                for (long i = 0; i < n; i++) {
+                    result = op.applyAsLong(result, vals.getByte(Integer.toUnsignedLong(ia.getInt(i))));
+                }
+            }
+            case LongArray la -> {
+                for (long i = 0; i < n; i++) {
+                    result = op.applyAsLong(result, vals.getByte(la.getLong(i)));
+                }
+            }
+            default -> throw new VortexException("DictByteArray: invalid codes type: "
+                    + codes.getClass().getSimpleName());
+        }
+        return result;
+    }
+}
diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/array/DictShortArray.java b/reader/src/main/java/io/github/dfa1/vortex/reader/array/DictShortArray.java
@@ -0,0 +1,150 @@
+package io.github.dfa1.vortex.reader.array;
+
+import io.github.dfa1.vortex.core.model.DType;
+import io.github.dfa1.vortex.core.error.VortexException;
+import io.github.dfa1.vortex.core.io.VortexFormat;
+
+import java.lang.foreign.MemorySegment;
+import java.lang.foreign.SegmentAllocator;
+import java.util.function.LongBinaryOperator;
+
+/// Dict-encoded [ShortArray] view. ADR 0012 shape.
+///
+/// Stores `values` (the dictionary pool) and `codes` (one index per
+/// row into `values`). Scalar access resolves on demand:
+/// `getShort(i) = values.getShort(codes.getCode(i))`. Per ADR 0012, this
+/// preserves zero-copy on dict-encoded categorical columns.
+///
+/// The `codes` array is typed as [Array] because the codes ptype
+/// varies with dictionary size — U8/U16/U32/U64 backed by
+/// [ByteArray]/[ShortArray]/[IntArray]/[LongArray].
+///
+/// @param dtype  logical element type (matches `values.dtype()`)
+/// @param length total logical row count (matches `codes.length()`)
+/// @param values dictionary pool — element at code `c` is `values.getShort(c)`
+/// @param codes  per-row index into `values`; must be one of
+///               [ByteArray], [ShortArray], [IntArray], [LongArray]
+public record DictShortArray(DType dtype, long length, ShortArray values, Array codes) implements ShortArray {
+
+    /// Builds a [DictShortArray], validating that `codes` is one of the
+    /// four narrow-int code array types and that its length matches `length`.
+    ///
+    /// @param dtype  logical element type
+    /// @param length total logical row count
+    /// @param values dictionary pool
+    /// @param codes  per-row code array (must be [ByteArray], [ShortArray],
+    ///               [IntArray], or [LongArray])
+    /// @return a new [DictShortArray]
+    /// @throws VortexException if `codes` is not a supported code-array type or
+    ///                         its length does not equal `length`
+    public static DictShortArray of(DType dtype, long length, ShortArray values, Array codes) {
+        DictArrays.validateCodes(codes, length);
+        return new DictShortArray(dtype, length, values, codes);
+    }
+
+    @Override
+    public short getShort(long i) {
+        return values.getShort(DictArrays.readCode(codes, i));
+    }
+
+    /// Materializes by gathering one dictionary value per code into a fresh
+    /// little-endian `i16` segment. The codes switch is hoisted outside the loop so
+    /// each branch is a uniform gather over a single code width.
+    ///
+    /// @param arena allocator for the output segment
+    /// @return a read-only little-endian `i16` segment of gathered values
+    /// @throws VortexException if `codes` is not a supported code-array type
+    @Override
+    public MemorySegment materialize(SegmentAllocator arena) {
+        long n = length;
+        MemorySegment dst = arena.allocate(n * 2L, 2);
+        ShortArray vals = values;
+        switch (codes) {
+            case ByteArray ba -> {
+                for (long i = 0; i < n; i++) {
+                    dst.setAtIndex(VortexFormat.LE_SHORT, i, vals.getShort(Byte.toUnsignedLong(ba.getByte(i))));
+                }
+            }
+            case ShortArray sa -> {
+                for (long i = 0; i < n; i++) {
+                    dst.setAtIndex(VortexFormat.LE_SHORT, i, vals.getShort(Short.toUnsignedLong(sa.getShort(i))));
+                }
+            }
+            case IntArray ia -> {
+                for (long i = 0; i < n; i++) {
+                    dst.setAtIndex(VortexFormat.LE_SHORT, i, vals.getShort(Integer.toUnsignedLong(ia.getInt(i))));
+                }
+            }
+            case LongArray la -> {
+                for (long i = 0; i < n; i++) {
+                    dst.setAtIndex(VortexFormat.LE_SHORT, i, vals.getShort(la.getLong(i)));
+                }
+            }
+            default -> throw new VortexException("DictShortArray: invalid codes type: "
+                    + codes.getClass().getSimpleName());
+        }
+        return dst.asReadOnly();
+    }
+
+    @Override
+    public void forEachShort(ShortConsumer cons) {
+        long n = length;
+        ShortArray vals = values;
+        switch (codes) {
+            case ByteArray ba -> {
+                for (long i = 0; i < n; i++) {
+                    cons.accept(vals.getShort(Byte.toUnsignedLong(ba.getByte(i))));
+                }
+            }
+            case ShortArray sa -> {
+                for (long i = 0; i < n; i++) {
+                    cons.accept(vals.getShort(Short.toUnsignedLong(sa.getShort(i))));
+                }
+            }
+            case IntArray ia -> {
+                for (long i = 0; i < n; i++) {
+                    cons.accept(vals.getShort(Integer.toUnsignedLong(ia.getInt(i))));
+                }
+            }
+            case LongArray la -> {
+                for (long i = 0; i < n; i++) {
+                    cons.accept(vals.getShort(la.getLong(i)));
+                }
+            }
+            default -> throw new VortexException("DictShortArray: invalid codes type: "
+                    + codes.getClass().getSimpleName());
+        }
+    }
+
+    @Override
+    public long fold(long identity, LongBinaryOperator op) {
+        long n = length;
+        ShortArray vals = values;
+        long result = identity;
+        switch (codes) {
+            case ByteArray ba -> {
+                for (long i = 0; i < n; i++) {
+                    result = op.applyAsLong(result, vals.getShort(Byte.toUnsignedLong(ba.getByte(i))));
+                }
+            }
+            case ShortArray sa -> {
+                for (long i = 0; i < n; i++) {
+                    result = op.applyAsLong(result, vals.getShort(Short.toUnsignedLong(sa.getShort(i))));
+                }
+            }
+            case IntArray ia -> {
+                for (long i = 0; i < n; i++) {
+                    result = op.applyAsLong(result, vals.getShort(Integer.toUnsignedLong(ia.getInt(i))));
+                }
+            }
+            case LongArray la -> {
+                for (long i = 0; i < n; i++) {
+                    result = op.applyAsLong(result, vals.getShort(la.getLong(i)));
+                }
+            }
+            default -> throw new VortexException("DictShortArray: invalid codes type: "
+                    + codes.getClass().getSimpleName());
+        }
+        return result;
+    }
+}
diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/layout/DictLayoutDecoder.java b/reader/src/main/java/io/github/dfa1/vortex/reader/layout/DictLayoutDecoder.java
@@ -10,14 +10,18 @@
 import io.github.dfa1.vortex.core.model.LayoutId;
 import io.github.dfa1.vortex.core.model.PType;
 import io.github.dfa1.vortex.reader.array.Array;
+import io.github.dfa1.vortex.reader.array.ByteArray;
+import io.github.dfa1.vortex.reader.array.DictByteArray;
 import io.github.dfa1.vortex.reader.array.DictDoubleArray;
 import io.github.dfa1.vortex.reader.array.DictFloatArray;
 import io.github.dfa1.vortex.reader.array.DictIntArray;
 import io.github.dfa1.vortex.reader.array.DictLongArray;
+import io.github.dfa1.vortex.reader.array.DictShortArray;
 import io.github.dfa1.vortex.reader.array.DoubleArray;
 import io.github.dfa1.vortex.reader.array.FloatArray;
 import io.github.dfa1.vortex.reader.array.IntArray;
 import io.github.dfa1.vortex.reader.array.LongArray;
+import io.github.dfa1.vortex.reader.array.ShortArray;
 import io.github.dfa1.vortex.reader.array.MaskedArray;
 import io.github.dfa1.vortex.reader.array.VarBinArray;
 
@@ -132,8 +136,11 @@ private static Array buildLazyDictPrimitive(DType.Primitive dtype, long n, Array
         return switch (ptype) {
             case I64, U64 -> DictLongArray.of(dtype, n, (LongArray) valuesData, codesData);
             case I32, U32 -> DictIntArray.of(dtype, n, (IntArray) valuesData, codesData);
+            case I16, U16 -> DictShortArray.of(dtype, n, (ShortArray) valuesData, codesData);
+            case I8, U8 -> DictByteArray.of(dtype, n, (ByteArray) valuesData, codesData);
             case F64 -> DictDoubleArray.of(dtype, n, (DoubleArray) valuesData, codesData);
             case F32 -> DictFloatArray.of(dtype, n, (FloatArray) valuesData, codesData);
+            // F16 has no Array subtype yet
             default -> throw new VortexException(EncodingId.VORTEX_DICT,
                     "layout: unsupported ptype for lazy dict: " + ptype);
         };
diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/array/DictRecordSmokeTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/array/DictRecordSmokeTest.java
