Add golden-corpus determinism tests for the default compressor#8697
Merged
Conversation
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | slice_empty_vortex |
310.3 ns | 368.6 ns | -15.83% |
| ❌ | Simulation | chunked_varbinview_opt_canonical_into[(100, 100)] |
305.4 µs | 340.4 µs | -10.27% |
| ⚡ | Simulation | chunked_varbinview_into_canonical[(1000, 10)] |
205.6 µs | 169.5 µs | +21.32% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing cost-model-r0-golden (29b6106) with develop (a649d75)
Footnotes
-
4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
connortsui20
added
the
changelog/skip
Claude
21 tasks
connortsui20
force-pushed
the
cost-model-r0-golden
branch
2 times, most recently
from
713fe12 to
00ccec3
Compare
connortsui20
force-pushed
the
cost-model-r0-golden
branch
from
00ccec3 to
1c1ad1b
Compare
Compress a fixed, seed-generated corpus covering every scheme's habitat (int monotone/sequence/low-cardinality/runs/sparse/null-heavy/negative/ random, ALP- and dict-friendly floats, FSST- and dict-friendly strings, binary, decimal, temporal, bool, struct, list) and snapshot each entry's full encoding tree with exact per-node byte counts via insta. Every entry is longer than 1024 values so the sampling-based estimation path is exercised, and each entry is compressed twice per run to assert determinism directly. These snapshots pin the default compressor's decisions: the upcoming cost-model refactor rungs cite zero snapshot churn as their proof of behavior preservation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Add two feature-gated golden variants over the same corpus: 'unstable' (unstable_encodings, default builder) pins Delta and Sequence selection, and 'compact' (unstable_encodings + zstd + pco, with_compact) pins Zstd and Pco selection. The default-variant test stays compiled out under unstable_encodings since that feature changes ALL_SCHEMES. A dedicated CI job (btrblocks-golden) runs each of the three feature combinations explicitly. The variants are mutually exclusive at compile time, so the existing --all-features workspace test jobs compile the default variant out and it would otherwise only run incidentally in the default-features musl job. OnPairScheme is excluded from both variants: its dictionary training (upstream onpair crate) iterates randomly-seeded hashbrown maps, so its compressed output — and therefore its sampled estimate — differs from run to run. A nondeterministic scheme cannot serve as a golden baseline, and it violates the determinism contract in the Scheme docs; that is a pre-existing bug worth its own issue, not something this suite can pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
insta snapshot files do not allow leading comment lines, so license them via a nested REUSE.toml, mirroring vortex-bench's handling of its own snapshots. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
cost-model-r0-golden
branch
from
1c1ad1b to
29b6106
Compare
robert3005
approved these changes
Jul 10, 2026
robert3005
left a comment
robert3005
left a comment
Contributor
There was a problem hiding this comment.
Not sure if insta is the right way of asserting it but don't have a better idea without inventing whole new format
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking Issue: #8701
Closes: #6171
Rationale for this change
Groundwork for making the compressor's selection policy pluggable (#7697): before any refactoring of scheme selection, pin the compressor's current decisions with reviewable golden snapshots, so each subsequent PR can demonstrate unchanged selection behavior by showing zero snapshot churn. Test-only, and lands first so later PRs diff against a committed baseline.
This also closes #6171 by adding dedicated regression coverage for whether the compressor produces the expected array tree, across a representative corpus rather than one-off encoding assertions.
What changes are included in this PR?
A golden test suite in
vortex-btrblocks: 20 seed-generated corpus entries spanning the registered schemes' habitats, each compressed twice per run (direct determinism check) and snapshotted as its full encoding tree with exact array-level byte counts. Three feature variants cover the default scheme pool,unstable_encodings, andwith_compact(+zstd+pco).A dedicated
btrblocks-goldenCI job runs all three feature combinations explicitly. The variants are mutually exclusive at compile time (unstable_encodingschangesALL_SCHEMES), so the existing--all-featuresworkspace test jobs compile thedefaultvariant out, and it would otherwise only run incidentally in the default-features musl job.OnPair is excluded from the golden compressors: its dictionary training is nondeterministic run-to-run (randomly-seeded hash maps in the upstream crate), so it cannot serve as a baseline. That's a pre-existing bug worth its own issue.
What APIs are changed? Are there any user-facing changes?
None (tests and CI only).