Raincloud triage round 268 269

.claude/skills/changelog.md

@@ -0,0 +1,64 @@
+---
+name: changelog
+description: Write CHANGELOG.md entries for unreleased commits — terse, one-line bullets that link to the introducing commit(s), not paragraphs. Triggers on "prepare the changelog", "update CHANGELOG", "write changelog entries", or invokes `/changelog`. Standalone version of the `release` skill's "CHANGELOG sync" step — use this when you want entries written without cutting a release yet.
+---
+
+## Overview
+
+Write CHANGELOG.md entries for commits that don't have one yet, under `## [Unreleased]`.
+Entries are **terse** — a title, not a paragraph — ending in a link to the introducing
+commit(s). The link is the receipt; anyone who wants the "why" and the numbers reads the
+commit message (`git show <sha>` or the GitHub commit page), not the changelog.
+
+## Steps
+
+1. Find the range of commits missing a changelog entry: compare `## [Unreleased]` in
+   `CHANGELOG.md` against `git log --oneline <last-documented-commit>..HEAD`. If unclear
+   where the last-documented commit is, ask.
+2. For each notable commit, write **one bullet**:
+   - A single short sentence — what changed, fewest words that stay precise. No numbers,
+     no "why", no before/after prose — those live in the commit message.
+   - End with the commit SHA(s), GitHub-auto-linked:
+     `([abc1234](https://github.com/dfa1/vortex-java/commit/abc1234))`
+   - One logical change spanning multiple commits: comma-separate SHAs, newest first.
+3. Categorize under the right `###` heading (create if missing): `Added` / `Changed` /
+   `Fixed` / `Performance` / `Removed` / `Security` — Keep-a-Changelog order (Added,
+   Changed, Deprecated, Removed, Fixed, Security), plus this project's `Performance`.
+4. Skip pure-internal churn: CI/Sonar fixes, source-tree moves, dedup refactors,
+   test/tooling-only commits (see [[feedback_changelog_terse]]) — omit rather than write a
+   bullet nobody user-facing cares about.
+5. Commit as `docs: CHANGELOG entries for <short description>`.
+
+## Example
+
+Good (terse — what to write):
+```
+- Utf8/Binary now use cost-based encoding selection instead of first-match dispatch. ([1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
+- FsstEncodingDecoder no longer throws on valid files with empty FSST metadata. ([1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
+```
+
+Bad (too verbose — do NOT write this):
+```
+- `CascadingCompressor` now runs `Utf8`/`Binary` columns through the same sample-and-measure
+  encoding competition `Primitive` columns already get, instead of first-match dispatch.
+  `DictEncodingEncoder` is registered ahead of `FsstEncodingEncoder`, so Dict always won
+  regardless of actual output size — confirmed by inspecting a 50k-row, high-cardinality
+  string file and finding `vortex.dict` where `vortex.fsst` is ~40% smaller. On
+  `highCardinalityUtf8_javaVsJni` (50k distinct 6-byte strings) this drops the Java/JNI
+  file-size ratio from 2.26× to 1.36× (801,908 → 483,336 bytes). ([1bbc3549](...))
+```
+The numbers and rationale belong in the commit message, not duplicated in the changelog.
+
+## Rules
+
+- Never invent entries for commits that don't exist — the link is the receipt.
+- Never leave a bullet without a commit link.
+- One sentence per bullet. If it needs two sentences, split it into two bullets, or the
+  extra detail belongs in the commit message instead.
+- Skip internal-only commits (matches the `release` skill's CHANGELOG sync policy).
+
+## When NOT to use
+
+- Finalizing a release's version header/date, or the compare link at the bottom of the
+  file — that's the `release` skill's job (its CHANGELOG sync step supersedes this one).
+- Writing release notes for work that isn't committed yet.

CHANGELOG.md

@@ -9,11 +9,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
-- Nullable low-cardinality columns now use the shared global dictionary across chunks instead of re-emitting a per-chunk dictionary. The writer previously excluded any nullable column from global-dict candidacy, so a nullable low-cardinality string/numeric column (extremely common in real data — most Raincloud categorical columns have nulls) re-wrote its dictionary in every chunk. Nullable columns now run the same cardinality/ratio gate against their non-null values and emit one shared `vortex.dict` layout with per-chunk masked codes; the reader combines codes-side and pool-side validity per row. On a 200k-row, 10-chunk nullable low-cardinality Utf8 column this shrinks the file 54% (232 KB → 107 KB) and closes the gap to the Rust reference from ~4× to ~1.5×. (internal)
-- `MaskedEncodingEncoder` now encodes an all-valid or all-invalid validity bitmap as `vortex.constant` instead of a raw per-row bitmap. A nullable column chunk with zero nulls previously still paid a full `n/8`-byte bitmap for no information; an all-valid or all-invalid chunk now costs a few bytes regardless of row count. (internal)
-- `MaskedEncodingEncoder` also tries `vortex.sparse` for a mixed-validity bitmap (fill = the majority value, patches = the minority positions) and keeps it over the raw bitmap when smaller — the patch-index array is compressed through a dedicated `vortex.sequence`/`fastlanes.delta`/FoR/bitpack cascade, so a clustered or regular null pattern (e.g. one null every 10 rows) collapses to a handful of bytes instead of a full bitmap. On the 200k-row, 10-chunk nullable low-cardinality Utf8 benchmark this drops the file a further 20% (107 KB → 86 KB) and the gap to the Rust reference from ~1.45× to ~1.17×. Also: `SequenceEncodingEncoder.encodeCascade` no longer lets `encode`'s "not an arithmetic sequence" exception escape when a stratified sample looked arithmetic but the full data wasn't — it now reports the cascade step as not applicable, like every other encoder's contract requires. (internal)
-- `ConstantEncodingEncoder` now accepts `DType.Bool` (previously `DType.Primitive` only), so any all-true or all-false dense Bool column — not just a masked/nullable one — can win `vortex.constant` through the normal per-column cascade, not only through `MaskedEncodingEncoder`'s dedicated check. `MaskedEncodingEncoder`'s validity encoding now delegates to it instead of duplicating the scalar-building logic. `RunEndEncodingEncoder` also gains a `vortex.runend` path for Bool: `MaskedEncodingEncoder` tries it alongside `vortex.sparse` and keeps whichever is smallest — `vortex.runend` wins on long clustered runs of valid/invalid rows (regardless of which value dominates), a shape `vortex.sparse`'s per-flip patch cost handles poorly. (internal)
-- `RleEncodingEncoder`/`RleEncodingDecoder` (`fastlanes.rle`) round out Bool coverage: the same FastLanes 1024-row chunked layout the numeric path uses, with a `vortex.bool`-encoded values pool (at most 2 distinct values per chunk) instead of a ptype-width one. `MaskedEncodingEncoder` tries it alongside `vortex.sparse`/`vortex.runend` and keeps the smallest — `DType.Bool` now has every general-purpose encoding family (`vortex.constant`, `vortex.sparse`, `vortex.runend`, `fastlanes.rle`) that `DType.Primitive` has. (internal)
+- A chunked `List` column spanning several flat chunks now decodes into a single stitched array instead of throwing a raw `ClassCastException`; any other unhandled dtype now fails with `VortexException`. ([#268](https://github.com/dfa1/vortex-java/issues/268))
+- A chunked `Utf8`/`Binary` column with an entirely-null chunk (`NullArray`) no longer throws `chunk is not a VarBinArray`; the chunk materializes as an all-null run. ([#269](https://github.com/dfa1/vortex-java/issues/269))
+- Nullable low-cardinality columns now share one global dictionary across chunks instead of re-emitting a per-chunk dictionary. ([5fe8b544](https://github.com/dfa1/vortex-java/commit/5fe8b544))
+- `MaskedEncodingEncoder` encodes an all-valid or all-invalid validity bitmap as `vortex.constant` instead of a raw per-row bitmap. ([ecd47ead](https://github.com/dfa1/vortex-java/commit/ecd47ead))
+- `MaskedEncodingEncoder` tries `vortex.sparse` for a mixed-validity bitmap and keeps it over the raw bitmap when smaller. ([506d036f](https://github.com/dfa1/vortex-java/commit/506d036f))
+- `SequenceEncodingEncoder.encodeCascade` no longer lets a full-data encode exception escape when a sampled prefix looked arithmetic but the full data wasn't. ([506d036f](https://github.com/dfa1/vortex-java/commit/506d036f))
+- `ConstantEncodingEncoder` now accepts `DType.Bool` (previously `Primitive` only). ([5379af66](https://github.com/dfa1/vortex-java/commit/5379af66))
+- `RunEndEncodingEncoder` gains a `vortex.runend` path for Bool. ([5379af66](https://github.com/dfa1/vortex-java/commit/5379af66))
+- `fastlanes.rle` now supports `DType.Bool`, completing Bool coverage across all general-purpose encodings (`vortex.constant`, `vortex.sparse`, `vortex.runend`, `fastlanes.rle`). ([fe6f132b](https://github.com/dfa1/vortex-java/commit/fe6f132b))
+- `Utf8`/`Binary` columns now use cost-based encoding selection instead of first-match dispatch (Dict no longer always beats FSST regardless of size). ([1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
+- `FsstEncodingDecoder` no longer throws on valid files with an all-default (empty) `vortex.fsst` metadata segment. ([1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
+- `FsstEncodingEncoder` writes symbol tables in the order the Rust reference requires; previously an out-of-order table could fail to open in the JNI/Rust reader. ([95e0dfb4](https://github.com/dfa1/vortex-java/commit/95e0dfb4))
+
+### Performance
+
+- `FsstEncodingEncoder`: iterative symbol-table training (variable-length 1–8 byte symbols, was a fixed 2-byte bigram scheme) and narrower metadata buffer ptypes. ([95e0dfb4](https://github.com/dfa1/vortex-java/commit/95e0dfb4), [1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
+- `FsstEncodingEncoder`: non-boxing symbol-match lookup and stratified training-sample selection. ([4a3fa0a9](https://github.com/dfa1/vortex-java/commit/4a3fa0a9))
 
 ## [0.12.2] — 2026-07-12
 

docs/compatibility.md

@@ -101,7 +101,7 @@ integer width is wire-legal, mirroring `VarBinArray`), and `ScanIterator` could
 | `vortex.listview`           | `ListViewEncodingDecoder`        | `ListViewEncodingEncoder`        | ✅      | ✅      |                                                                       |
 | `vortex.fixed_size_list`    | `FixedSizeListEncodingDecoder`   | `FixedSizeListEncodingEncoder`   | ✅      | ✅      |                                                                       |
 | `vortex.zstd`               | `ZstdEncodingDecoder`            | `ZstdEncodingEncoder`            | ✅      | ✅      | Primitive, Utf8, Binary                                               |
-| `vortex.masked`             | `MaskedEncodingDecoder`          | `MaskedEncodingEncoder`          | ✅      | ✅      | NullableData carrier; inner values cascade (Dict/FSST/ALP/…) when cascading is enabled, else first-match Primitive / VarBin / FixedSizeList |
+| `vortex.masked`             | `MaskedEncodingDecoder`          | `MaskedEncodingEncoder`          | ✅      | ✅      | NullableData carrier; inner values cascade (Dict/FSST/ALP/…) when cascading is enabled; Utf8/Primitive still compete on measured size even at depth 0, else first-match FixedSizeList |
 | `vortex.decimal`            | `DecimalEncodingDecoder`         | `DecimalEncodingEncoder`         | ✅      | ✅      |                                                                       |
 | `vortex.decimal_byte_parts` | `DecimalBytePartsEncodingDecoder`| `DecimalBytePartsEncodingEncoder`| ✅      | ✅      |                                                                       |
 | `vortex.datetimeparts`      | `DateTimePartsEncodingDecoder`   | `DateTimePartsEncodingEncoder`   | ✅      | ✅      |                                                                       |

integration/src/test/java/io/github/dfa1/vortex/integration/FileSizeComparisonIntegrationTest.java

@@ -620,11 +620,11 @@ void highCardinalityUtf8_javaVsJni(@TempDir Path tmp) throws IOException {
                 (double) javaSize / jniSize,
                 (double) javaSize / rawBytes);
 
-        // Then — Java within 2.5x of JNI. Java's FSST symbol-table builder is still less
-        // aggressive than Rust's on truly random short strings (Linux Rust FSST ≈2.26x
-        // observed); tighten further when FsstEncoding.Encoder gets iterative symbol
-        // training (per the FSST paper).
-        assertThat((double) javaSize / jniSize).isLessThan(2.5);
+        // Then — Java within 1.5x of JNI (observed ≈1.36x). CascadingCompressor now runs Utf8
+        // through the same sample-and-measure competition as Primitive dtypes instead of
+        // first-match dispatch, so FSST — not Dict — wins on this high-cardinality column
+        // (Dict's table for ~50k distinct 6-byte values used to dominate the file size).
+        assertThat((double) javaSize / jniSize).isLessThan(1.5);
 
         // Then — Java file is readable and row count matches
         var totalRows = new java.util.concurrent.atomic.AtomicLong();

integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java

@@ -569,7 +569,7 @@ void javaWriter_jniReader_utf8Column(@TempDir Path tmp) throws IOException {
 
     @Test
     void javaWriter_jniReader_fsstUtf8Column(@TempDir Path tmp) throws IOException {
-        // Given — FSST encoding: bigram symbol table, escape fallback for unmatched bytes
+        // Given — FSST encoding: trained variable-length symbol table, escape fallback for unmatched bytes
         Path file = tmp.resolve("java_fsst_utf8.vtx");
         String[] data = {"apple", "banana", "cherry", "apricot", "avocado", "almond"};
         try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);

integration/src/test/java/io/github/dfa1/vortex/integration/RaincloudConformanceIntegrationTest.java

@@ -87,7 +87,11 @@ Stream<DynamicTest> conformancePerSlug() throws IOException {
                     String status = expected.getOrDefault(slug, "untriaged");
                     switch (status) {
                         case "ok" -> assertMatchesParquetOracle(vortex, parquet);
-                        case "untriaged" -> reportUntriaged(vortex, parquet);
+                        // missing_auth slugs never reach here in practice — hydration fails
+                        // before a manifest entry exists — but report the same as untriaged
+                        // rather than falling through to assertStillFails (wrong: that path
+                        // means "known reader gap", not "couldn't even attempt this").
+                        case "untriaged", "missing_auth" -> reportUntriaged(vortex, parquet);
                         default -> assertStillFails(vortex, parquet, status);
                     }
                 }));
@@ -362,6 +366,12 @@ private static String oracleCell(SchemaNode node, RowReader rows, int fieldIndex
             if (group.isMap()) {
                 throw new TestAbortedException("oracle cannot format MAP column: " + group.name());
             }
+            // hardwood's own NestedRowReader.getStruct() mis-casts a VARIANT-shredded group
+            // (metadata/value/typed_value) to its plain-Struct FieldDesc and throws
+            // ClassCastException — an oracle bug, not a vortex-java gap. Abort before hitting it.
+            if (group.isVariant()) {
+                throw new TestAbortedException("oracle cannot format VARIANT column: " + group.name());
+            }
             return oracleJsonObject(group, rows.getStruct(fieldIndex));
         }
         SchemaNode.PrimitiveNode prim = (SchemaNode.PrimitiveNode) node;