Raincloud triage round 268 269#271
Open
dfa1 wants to merge 11 commits into
Open
Conversation
Replace the fixed 2-byte bigram symbol table with iterative FSST-paper-style training (longest-match-first parsing, gain-ranked candidate refinement) producing variable-length 1-8 byte symbols. Order the final table per the Rust reference's wire contract (FSSTData::validate_symbol_lengths): multi-byte symbols non-decreasing by length, then all length-1 symbols, verified against the JNI reader. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the per-symbol-table HashMap<SymbolCandidate, Integer> match index with a flat open-addressing table over primitive arrays. longestMatch() runs up to 8 probes per byte position across every row in the dataset; boxing a record key per probe was measurable allocation pressure in the hottest loop this encoder has. ~18-23% faster on incompressible data (worst case: many failed probes), ~10% on realistic repetitive text. Also replace first-N training sampling with the same stratified, seeded-PRNG scheme CascadingCompressor already uses, so training doesn't silently bias toward whatever sorts first in the input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of the Java/JNI file-size gap on high-cardinality string
columns: CascadingCompressor routed Utf8/Binary through first-match
dispatch (findPrimitiveEncoding), not the sample-and-measure
competition Primitive dtypes get. Dict is registered before FSST, so
Dict always won regardless of actual size — confirmed by inspecting a
50k-distinct-string file and finding vortex.dict where FSST would be
~40% smaller. Both DictEncodingEncoder's own comment ("Utf8 still uses
sample-encoded selection") and VortexWriter's DEFAULT_CODECS comment
already documented this as the intended behavior; it just wasn't wired
up.
Fix: extract the shared skip/always-use sweep + stratified-sample
measurement loop into competeAndEncode(), reuse it for both Primitive
(unchanged, using primitiveBytes as baseline) and the new Utf8 path
(no analytic baseline available, so the loop just keeps the smallest
measured candidate — VarBinEncodingEncoder unconditionally accepts
Utf8 so a winner always exists). stratifiedSample/dataLength gain a
String[] case to support sampling string columns.
Also narrow FsstEncodingEncoder's uncompressed_lengths/codes_offsets
buffers to the smallest ptype that fits (was always hardcoded I32),
using the metadata fields the wire format already carries for this.
This surfaced a latent bug: proto3 omits fields at their default
(zero/U8) value, so an all-U8 FSST metadata message encodes to zero
bytes and the writer skips the metadata segment entirely. Absent
metadata is a valid encoding of that default, not corruption —
FsstEncodingDecoder's unconditional null-check threw on legitimate
files. Fixed to treat null metadata as defaults, and added proper
buffer/child-count bounds checks (previously null-metadata was
incidentally catching a separate malformed-input case that needed its
own explicit guard).
Net effect on highCardinalityUtf8_javaVsJni (50k distinct 6-byte
strings): Java/JNI 2.26x -> 1.36x (801,908 -> 483,336 bytes).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ai2-arc, amazon-reviews-2023-subscription-boxes, anthropic-economic-index, anthropic-hh-rlhf-helpful-base, anthropic-interviewer, aya-collection-templated, aya-dataset all match the Parquet oracle exactly — no gaps found. 3 remaining candidates (airbnb-prices-in-european-cities, airbnbopendata, bank-account-fraud-dataset-neurips-2022) are Kaggle-hosted and need Kaggle API credentials this environment doesn't have; left untriaged. 117 ok/130 untriaged -> 124 ok/123 untriaged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Comment lines above the 3 Kaggle-hosted slugs from this round (airbnb-prices-in-european-cities, airbnbopendata, bank-account-fraud-dataset-neurips-2022) noting they need Kaggle API credentials to hydrate — distinguishes "blocked on creds" from "just not attempted yet" for future triagers. Status stays literal "untriaged" since the harness string-matches it exactly; comments are skipped by the parser. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the ad-hoc "#-comment above an untriaged line" convention with
a proper matrix status: missing_auth marks a slug that can't even be
hydrated because its source (Kaggle, gated HuggingFace, ...) needs
credentials this environment doesn't have. Distinguishes "blocked on
a credential" from plain untriaged ("just hasn't been attempted") so
future triagers don't waste time rediscovering the same blocker.
Harness treats it identically to untriaged (reportUntriaged, not
assertStillFails — that path means "known reader gap", which this
isn't). Applied to the 3 Kaggle-blocked slugs from the last round.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cover this session's production-code changes not yet documented: Utf8/Binary cost-based dispatch fix, FsstEncodingDecoder null-metadata bug, FSST wire symbol-table ordering fix, and the FSST encoder's iterative training + non-boxing lookup + ptype narrowing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The 5 entries I added for the FSST/dispatch work were verbose paragraphs (full rationale + before/after numbers inline) — too verbose per feedback. Rewrote them as one-sentence titles ending in a commit link; the numbers and rationale live in the commit message, which the link points to. Also records a standalone `changelog` skill (.claude/skills/) codifying this convention, split out of the `release` skill's CHANGELOG-sync step so it can run without cutting a release. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rewrite the 5 remaining paragraph-style Unreleased entries (nullable global dict, Bool constant/sparse/runend/rle) as one-sentence titles with commit links, matching the new changelog skill. Splits the combined Sparse+Sequence and Constant+RunEnd bullets into one entry per logical change (same source commit, distinct fixes). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ChunkedLayoutDecoder.decodeChunkedLayout blindly cast dtype to DType.Primitive for anything that wasn't Bool/Utf8/Binary, throwing a raw ClassCastException instead of VortexException for chunked List columns (#268). VarBinArray.ChunkedMode.of separately threw on a chunk that decoded to NullArray (an all-null vortex.null/vortex.constant chunk) instead of a VarBinArray (#269). Extracts the per-chunk dispatch into ChunkedArrayCombiner: primitives and Bool keep their existing ChunkedXxxArray shapes, Utf8/Binary route through VarBinArray.ChunkedMode (now NullArray-aware, materializing an all-null chunk as a zero-length OffsetMode run), List columns stitch into one ListArray via a recursive combine of their element arrays plus a rebuilt cumulative offsets table, and any other dtype now fails with a VortexException instead of a raw cast. combineChunkValidity also folds NullArray chunks into the row-validity bitmap it reconstructs across chunks. Found via the Raincloud conformance corpus (squad-v2, oasst1, mnist). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Oracle guard: hardwood's NestedRowReader.getStruct() mis-casts a
VARIANT-shredded group (metadata/value/typed_value) to its plain
Struct FieldDesc and throws ClassCastException — an oracle bug, not
a vortex-java gap. RaincloudConformanceIntegrationTest now aborts on
group.isVariant() before hitting it, same as the existing isMap()
guard (unblocks countries-of-the-world, which regressed to a hard
failure once the corpus was rebuilt with a VARIANT column).
- Triage: 32 untriaged slugs verified ok against the parquet oracle
(13 large BI slugs + 19 smaller HF/benchmark slugs). squad-v2 and
mnist flip from gap:268 to ok now that the previous commit's fix
lands. oasst1 hits a different, new bug — a null Struct row renders
as {"field":null,...} instead of an empty cell (#270, filed) — so it
moves from gap:268 to gap:270 rather than to ok.
Matrix: 124 ok / 0 gaps / 123 untriaged -> 158 ok / 1 gap / 85 untriaged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.