You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Rewrite the 5 remaining paragraph-style Unreleased entries (nullable
global dict, Bool constant/sparse/runend/rle) as one-sentence titles
with commit links, matching the new changelog skill. Splits the
combined Sparse+Sequence and Constant+RunEnd bullets into one entry
per logical change (same source commit, distinct fixes).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: CHANGELOG.md
+7-5Lines changed: 7 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,11 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
9
9
10
10
### Fixed
11
11
12
-
- Nullable low-cardinality columns now use the shared global dictionary across chunks instead of re-emitting a per-chunk dictionary. The writer previously excluded any nullable column from global-dict candidacy, so a nullable low-cardinality string/numeric column (extremely common in real data — most Raincloud categorical columns have nulls) re-wrote its dictionary in every chunk. Nullable columns now run the same cardinality/ratio gate against their non-null values and emit one shared `vortex.dict` layout with per-chunk masked codes; the reader combines codes-side and pool-side validity per row. On a 200k-row, 10-chunk nullable low-cardinality Utf8 column this shrinks the file 54% (232 KB → 107 KB) and closes the gap to the Rust reference from ~4× to ~1.5×. (internal)
13
-
-`MaskedEncodingEncoder` now encodes an all-valid or all-invalid validity bitmap as `vortex.constant` instead of a raw per-row bitmap. A nullable column chunk with zero nulls previously still paid a full `n/8`-byte bitmap for no information; an all-valid or all-invalid chunk now costs a few bytes regardless of row count. (internal)
14
-
-`MaskedEncodingEncoder` also tries `vortex.sparse` for a mixed-validity bitmap (fill = the majority value, patches = the minority positions) and keeps it over the raw bitmap when smaller — the patch-index array is compressed through a dedicated `vortex.sequence`/`fastlanes.delta`/FoR/bitpack cascade, so a clustered or regular null pattern (e.g. one null every 10 rows) collapses to a handful of bytes instead of a full bitmap. On the 200k-row, 10-chunk nullable low-cardinality Utf8 benchmark this drops the file a further 20% (107 KB → 86 KB) and the gap to the Rust reference from ~1.45× to ~1.17×. Also: `SequenceEncodingEncoder.encodeCascade` no longer lets `encode`'s "not an arithmetic sequence" exception escape when a stratified sample looked arithmetic but the full data wasn't — it now reports the cascade step as not applicable, like every other encoder's contract requires. (internal)
15
-
-`ConstantEncodingEncoder` now accepts `DType.Bool` (previously `DType.Primitive` only), so any all-true or all-false dense Bool column — not just a masked/nullable one — can win `vortex.constant` through the normal per-column cascade, not only through `MaskedEncodingEncoder`'s dedicated check. `MaskedEncodingEncoder`'s validity encoding now delegates to it instead of duplicating the scalar-building logic. `RunEndEncodingEncoder` also gains a `vortex.runend` path for Bool: `MaskedEncodingEncoder` tries it alongside `vortex.sparse` and keeps whichever is smallest — `vortex.runend` wins on long clustered runs of valid/invalid rows (regardless of which value dominates), a shape `vortex.sparse`'s per-flip patch cost handles poorly. (internal)
16
-
-`RleEncodingEncoder`/`RleEncodingDecoder` (`fastlanes.rle`) round out Bool coverage: the same FastLanes 1024-row chunked layout the numeric path uses, with a `vortex.bool`-encoded values pool (at most 2 distinct values per chunk) instead of a ptype-width one. `MaskedEncodingEncoder` tries it alongside `vortex.sparse`/`vortex.runend` and keeps the smallest — `DType.Bool` now has every general-purpose encoding family (`vortex.constant`, `vortex.sparse`, `vortex.runend`, `fastlanes.rle`) that `DType.Primitive` has. (internal)
12
+
- Nullable low-cardinality columns now share one global dictionary across chunks instead of re-emitting a per-chunk dictionary. ([5fe8b544](https://github.com/dfa1/vortex-java/commit/5fe8b544))
13
+
-`MaskedEncodingEncoder` encodes an all-valid or all-invalid validity bitmap as `vortex.constant` instead of a raw per-row bitmap. ([ecd47ead](https://github.com/dfa1/vortex-java/commit/ecd47ead))
14
+
-`MaskedEncodingEncoder` tries `vortex.sparse` for a mixed-validity bitmap and keeps it over the raw bitmap when smaller. ([506d036f](https://github.com/dfa1/vortex-java/commit/506d036f))
15
+
-`SequenceEncodingEncoder.encodeCascade` no longer lets a full-data encode exception escape when a sampled prefix looked arithmetic but the full data wasn't. ([506d036f](https://github.com/dfa1/vortex-java/commit/506d036f))
16
+
-`ConstantEncodingEncoder` now accepts `DType.Bool` (previously `Primitive` only). ([5379af66](https://github.com/dfa1/vortex-java/commit/5379af66))
17
+
-`RunEndEncodingEncoder` gains a `vortex.runend` path for Bool. ([5379af66](https://github.com/dfa1/vortex-java/commit/5379af66))
18
+
-`fastlanes.rle` now supports `DType.Bool`, completing Bool coverage across all general-purpose encodings (`vortex.constant`, `vortex.sparse`, `vortex.runend`, `fastlanes.rle`). ([fe6f132b](https://github.com/dfa1/vortex-java/commit/fe6f132b))
17
19
-`Utf8`/`Binary` columns now use cost-based encoding selection instead of first-match dispatch (Dict no longer always beats FSST regardless of size). ([1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
18
20
-`FsstEncodingDecoder` no longer throws on valid files with an all-default (empty) `vortex.fsst` metadata segment. ([1bbc3549](https://github.com/dfa1/vortex-java/commit/1bbc3549))
19
21
-`FsstEncodingEncoder` writes symbol tables in the order the Rust reference requires; previously an out-of-order table could fail to open in the JNI/Rust reader. ([95e0dfb4](https://github.com/dfa1/vortex-java/commit/95e0dfb4))
0 commit comments