Replace the per-symbol-table HashMap<SymbolCandidate, Integer> match
index with a flat open-addressing table over primitive arrays.
longestMatch() runs up to 8 probes per byte position across every row
in the dataset; boxing a record key per probe was measurable
allocation pressure in the hottest loop this encoder has.
~18-23% faster on incompressible data (worst case: many failed
probes), ~10% on realistic repetitive text.

Also replace first-N training sampling with the same stratified,
seeded-PRNG scheme CascadingCompressor already uses, so training
doesn't silently bias toward whatever sorts first in the input.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Root cause of the Java/JNI file-size gap on high-cardinality string
columns: CascadingCompressor routed Utf8/Binary through first-match
dispatch (findPrimitiveEncoding), not the sample-and-measure
competition Primitive dtypes get. Dict is registered before FSST, so
Dict always won regardless of actual size — confirmed by inspecting a
50k-distinct-string file and finding vortex.dict where FSST would be
~40% smaller. Both DictEncodingEncoder's own comment ("Utf8 still uses
sample-encoded selection") and VortexWriter's DEFAULT_CODECS comment
already documented this as the intended behavior; it just wasn't wired
up.

Fix: extract the shared skip/always-use sweep + stratified-sample
measurement loop into competeAndEncode(), reuse it for both Primitive
(unchanged, using primitiveBytes as baseline) and the new Utf8 path
(no analytic baseline available, so the loop just keeps the smallest
measured candidate — VarBinEncodingEncoder unconditionally accepts
Utf8 so a winner always exists). stratifiedSample/dataLength gain a
String[] case to support sampling string columns.

Also narrow FsstEncodingEncoder's uncompressed_lengths/codes_offsets
buffers to the smallest ptype that fits (was always hardcoded I32),
using the metadata fields the wire format already carries for this.

This surfaced a latent bug: proto3 omits fields at their default
(zero/U8) value, so an all-U8 FSST metadata message encodes to zero
bytes and the writer skips the metadata segment entirely. Absent
metadata is a valid encoding of that default, not corruption —
FsstEncodingDecoder's unconditional null-check threw on legitimate
files. Fixed to treat null metadata as defaults, and added proper
buffer/child-count bounds checks (previously null-metadata was
incidentally catching a separate malformed-input case that needed its
own explicit guard).

Net effect on highCardinalityUtf8_javaVsJni (50k distinct 6-byte
strings): Java/JNI 2.26x -> 1.36x (801,908 -> 483,336 bytes).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ai2-arc, amazon-reviews-2023-subscription-boxes, anthropic-economic-index,
anthropic-hh-rlhf-helpful-base, anthropic-interviewer, aya-collection-templated,
aya-dataset all match the Parquet oracle exactly — no gaps found.

3 remaining candidates (airbnb-prices-in-european-cities, airbnbopendata,
bank-account-fraud-dataset-neurips-2022) are Kaggle-hosted and need
Kaggle API credentials this environment doesn't have; left untriaged.

117 ok/130 untriaged -> 124 ok/123 untriaged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Comment lines above the 3 Kaggle-hosted slugs from this round
(airbnb-prices-in-european-cities, airbnbopendata,
bank-account-fraud-dataset-neurips-2022) noting they need Kaggle API
credentials to hydrate — distinguishes "blocked on creds" from
"just not attempted yet" for future triagers. Status stays literal
"untriaged" since the harness string-matches it exactly; comments are
skipped by the parser.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Replace the ad-hoc "#-comment above an untriaged line" convention with
a proper matrix status: missing_auth marks a slug that can't even be
hydrated because its source (Kaggle, gated HuggingFace, ...) needs
credentials this environment doesn't have. Distinguishes "blocked on
a credential" from plain untriaged ("just hasn't been attempted") so
future triagers don't waste time rediscovering the same blocker.

Harness treats it identically to untriaged (reportUntriaged, not
assertStillFails — that path means "known reader gap", which this
isn't). Applied to the 3 Kaggle-blocked slugs from the last round.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Cover this session's production-code changes not yet documented:
Utf8/Binary cost-based dispatch fix, FsstEncodingDecoder null-metadata
bug, FSST wire symbol-table ordering fix, and the FSST encoder's
iterative training + non-boxing lookup + ptype narrowing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The 5 entries I added for the FSST/dispatch work were verbose
paragraphs (full rationale + before/after numbers inline) — too
verbose per feedback. Rewrote them as one-sentence titles ending in
a commit link; the numbers and rationale live in the commit message,
which the link points to.

Also records a standalone `changelog` skill (.claude/skills/) codifying
this convention, split out of the `release` skill's CHANGELOG-sync step
so it can run without cutting a release.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Rewrite the 5 remaining paragraph-style Unreleased entries (nullable
global dict, Bool constant/sparse/runend/rle) as one-sentence titles
with commit links, matching the new changelog skill. Splits the
combined Sparse+Sequence and Constant+RunEnd bullets into one entry
per logical change (same source commit, distinct fixes).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ChunkedLayoutDecoder.decodeChunkedLayout blindly cast dtype to
DType.Primitive for anything that wasn't Bool/Utf8/Binary, throwing a
raw ClassCastException instead of VortexException for chunked List
columns (#268). VarBinArray.ChunkedMode.of separately threw on a chunk
that decoded to NullArray (an all-null vortex.null/vortex.constant
chunk) instead of a VarBinArray (#269).

Extracts the per-chunk dispatch into ChunkedArrayCombiner: primitives
and Bool keep their existing ChunkedXxxArray shapes, Utf8/Binary route
through VarBinArray.ChunkedMode (now NullArray-aware, materializing an
all-null chunk as a zero-length OffsetMode run), List columns stitch
into one ListArray via a recursive combine of their element arrays plus
a rebuilt cumulative offsets table, and any other dtype now fails with
a VortexException instead of a raw cast. combineChunkValidity also
folds NullArray chunks into the row-validity bitmap it reconstructs
across chunks.

Found via the Raincloud conformance corpus (squad-v2, oasst1, mnist).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

- Oracle guard: hardwood's NestedRowReader.getStruct() mis-casts a
  VARIANT-shredded group (metadata/value/typed_value) to its plain
  Struct FieldDesc and throws ClassCastException — an oracle bug, not
  a vortex-java gap. RaincloudConformanceIntegrationTest now aborts on
  group.isVariant() before hitting it, same as the existing isMap()
  guard (unblocks countries-of-the-world, which regressed to a hard
  failure once the corpus was rebuilt with a VARIANT column).
- Triage: 32 untriaged slugs verified ok against the parquet oracle
  (13 large BI slugs + 19 smaller HF/benchmark slugs). squad-v2 and
  mnist flip from gap:268 to ok now that the previous commit's fix
  lands. oasst1 hits a different, new bug — a null Struct row renders
  as {"field":null,...} instead of an empty cell (#270, filed) — so it
  moves from gap:268 to gap:270 rather than to ok.

Matrix: 124 ok / 0 gaps / 123 untriaged -> 158 ok / 1 gap / 85 untriaged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>