Skip to content

Commit eec25c1

Browse files
dfa1claude
andcommitted
docs: CHANGELOG entries for FSST/dispatch work
Cover this session's production-code changes not yet documented: Utf8/Binary cost-based dispatch fix, FsstEncodingDecoder null-metadata bug, FSST wire symbol-table ordering fix, and the FSST encoder's iterative training + non-boxing lookup + ptype narrowing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent b081a17 commit eec25c1

1 file changed

Lines changed: 8 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1414
- `MaskedEncodingEncoder` also tries `vortex.sparse` for a mixed-validity bitmap (fill = the majority value, patches = the minority positions) and keeps it over the raw bitmap when smaller — the patch-index array is compressed through a dedicated `vortex.sequence`/`fastlanes.delta`/FoR/bitpack cascade, so a clustered or regular null pattern (e.g. one null every 10 rows) collapses to a handful of bytes instead of a full bitmap. On the 200k-row, 10-chunk nullable low-cardinality Utf8 benchmark this drops the file a further 20% (107 KB → 86 KB) and the gap to the Rust reference from ~1.45× to ~1.17×. Also: `SequenceEncodingEncoder.encodeCascade` no longer lets `encode`'s "not an arithmetic sequence" exception escape when a stratified sample looked arithmetic but the full data wasn't — it now reports the cascade step as not applicable, like every other encoder's contract requires. (internal)
1515
- `ConstantEncodingEncoder` now accepts `DType.Bool` (previously `DType.Primitive` only), so any all-true or all-false dense Bool column — not just a masked/nullable one — can win `vortex.constant` through the normal per-column cascade, not only through `MaskedEncodingEncoder`'s dedicated check. `MaskedEncodingEncoder`'s validity encoding now delegates to it instead of duplicating the scalar-building logic. `RunEndEncodingEncoder` also gains a `vortex.runend` path for Bool: `MaskedEncodingEncoder` tries it alongside `vortex.sparse` and keeps whichever is smallest — `vortex.runend` wins on long clustered runs of valid/invalid rows (regardless of which value dominates), a shape `vortex.sparse`'s per-flip patch cost handles poorly. (internal)
1616
- `RleEncodingEncoder`/`RleEncodingDecoder` (`fastlanes.rle`) round out Bool coverage: the same FastLanes 1024-row chunked layout the numeric path uses, with a `vortex.bool`-encoded values pool (at most 2 distinct values per chunk) instead of a ptype-width one. `MaskedEncodingEncoder` tries it alongside `vortex.sparse`/`vortex.runend` and keeps the smallest — `DType.Bool` now has every general-purpose encoding family (`vortex.constant`, `vortex.sparse`, `vortex.runend`, `fastlanes.rle`) that `DType.Primitive` has. (internal)
17+
- `CascadingCompressor` now runs `Utf8`/`Binary` columns through the same sample-and-measure encoding competition `Primitive` columns already get, instead of first-match dispatch. `DictEncodingEncoder` is registered ahead of `FsstEncodingEncoder`, so Dict always won regardless of actual output size — confirmed by inspecting a 50k-row, high-cardinality string file and finding `vortex.dict` where `vortex.fsst` is ~40% smaller. On `highCardinalityUtf8_javaVsJni` (50k distinct 6-byte strings) this drops the Java/JNI file-size ratio from 2.26× to 1.36× (801,908 → 483,336 bytes). ([1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
18+
- `FsstEncodingDecoder` no longer throws on a valid file whose `vortex.fsst` metadata segment is absent. Proto3 omits fields at their default (zero/`U8`) value, so an all-`U8` metadata message legitimately encodes to zero bytes and the writer omits the segment entirely; the decoder previously treated any absent metadata as corruption. Also adds explicit buffer-count (3) and child-count (≥2) bounds checks, which the removed null-metadata check had been incidentally — and incorrectly — covering for a separate malformed-input case. ([1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
19+
- `FsstEncodingEncoder` now writes its symbol table in the order the Rust reference requires (`FSSTData::validate_symbol_lengths`: multi-byte symbols non-decreasing by length, then all length-1 symbols) — a Java-written file with an out-of-order table previously failed to open in the JNI/Rust reader. ([95e0dfb4](https://github.com/dfa1/vortex-java/commit/95e0dfb4))
20+
21+
### Performance
22+
23+
- `FsstEncodingEncoder` replaces its fixed 2-byte bigram symbol table with iterative FSST-paper-style training (longest-match-first parsing, gain-ranked candidate refinement over several passes), producing variable-length 1–8 byte symbols instead of a fixed 2-byte cap. Also narrows the `uncompressed_lengths`/`codes_offsets` metadata buffers to the smallest ptype that fits the actual data (previously always 4-byte `I32`) using wire-format fields that already existed for this. ([95e0dfb4](https://github.com/dfa1/vortex-java/commit/95e0dfb4), [1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
24+
- `FsstEncodingEncoder`'s symbol-match lookup replaces a `HashMap<SymbolCandidate, Integer>` (boxing a record key on every probe) with a flat open-addressing table over primitive arrays; training sampling switches from "first N rows" to the same stratified stride sampling `CascadingCompressor` uses elsewhere, avoiding bias on sorted/clustered input. ~18–23% faster encode on incompressible data, ~10% on repetitive text. ([4a3fa0a9](https://github.com/dfa1/vortex-java/commit/4a3fa0a9))
1725

1826
## [0.12.2] — 2026-07-12
1927

0 commit comments

Comments
 (0)