Generalize the deferred-estimation early-exit to the cost model#8699
Draft
connortsui20 wants to merge 8 commits into
Draft
Generalize the deferred-estimation early-exit to the cost model#8699connortsui20 wants to merge 8 commits into
connortsui20 wants to merge 8 commits into
Conversation
Merging this PR will degrade performance by 10.24%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
21 tasks
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
from
e39c549 to
e17a967
Compare
connortsui20
force-pushed
the
cost-model-r3-lower-bound
branch
from
56f8b12 to
f4617b8
Compare
Compress a fixed, seed-generated corpus covering every scheme's habitat (int monotone/sequence/low-cardinality/runs/sparse/null-heavy/negative/ random, ALP- and dict-friendly floats, FSST- and dict-friendly strings, binary, decimal, temporal, bool, struct, list) and snapshot each entry's full encoding tree with exact per-node byte counts via insta. Every entry is longer than 1024 values so the sampling-based estimation path is exercised, and each entry is compressed twice per run to assert determinism directly. These snapshots pin the default compressor's decisions: the upcoming cost-model refactor rungs cite zero snapshot churn as their proof of behavior preservation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Add two feature-gated golden variants over the same corpus: 'unstable' (unstable_encodings, default builder) pins Delta and Sequence selection, and 'compact' (unstable_encodings + zstd + pco, with_compact) pins Zstd and Pco selection. The default-variant test stays compiled out under unstable_encodings since that feature changes ALL_SCHEMES. A dedicated CI job (btrblocks-golden) runs each of the three feature combinations explicitly. The variants are mutually exclusive at compile time, so the existing --all-features workspace test jobs compile the default variant out and it would otherwise only run incidentally in the default-features musl job. OnPairScheme is excluded from both variants: its dictionary training (upstream onpair crate) iterates randomly-seeded hashbrown maps, so its compressed output — and therefore its sampled estimate — differs from run to run. A nondeterministic scheme cannot serve as a golden baseline, and it violates the determinism contract in the Scheme docs; that is a pre-existing bug worth its own issue, not something this suite can pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
insta snapshot files do not allow leading comment lines, so license them via a nested REUSE.toml, mirroring vortex-bench's handling of its own snapshots. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Pure refactor of choose_best_scheme and the sampling estimator: - estimate_compression_ratio_with_sampling now returns a SampledEstimate carrying the compressed sample array alongside its score, instead of measuring nbytes and dropping the array. - Selection bookkeeping switches from (scheme, EstimateScore) tuples to a crate-private Candidate carrying the mechanical facts about each option (scheme, score, input bytes, value count, sampled array, cascade ancestry). Candidates are still compared by ratio, ties still break by registration order, and the sampled array is dropped at the end of selection, so behavior is unchanged: same winners, same estimates, same output (golden suite has zero snapshot churn). The not-yet-read Candidate fields are the facts a pluggable cost model prices in the follow-up rungs (#7697) with no new Scheme API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
New vortex_compressor::cost module extracting selection policy into a pluggable model, per the compressor cost-model plan (#7697): - Cost: opaque finite f64 newtype, lower is better; units are defined per model since costs are only compared within one model. - CostModel: price a Candidate (None = reject) and price the canonical baseline every candidate must strictly beat. The module docs state the axioms that stay outside the model: AlwaysUse short-circuiting and the byte-acceptance gate (including its AnyScalarFn carve-out). - SizeCost: prices a candidate at Cost(-ratio) over the exact ratio signal and canonical at Cost(-1.0), reproducing ratio-argmax selection bit-exactly. Pricing off the ratio rather than recomputed bytes (input/ratio) avoids IEEE division collapsing strict ratio inequalities into ties and flipping winners via registration order. Candidate becomes public (it is the trait's argument), re-exported via the cost module. Unit tests pin the validity gate, pairwise ordering and tie behavior against the historical rules. Nothing consumes the module yet; the selection loop is wired to it in the next commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
choose_best_scheme now prices every scored candidate through the
compressor's CostModel: the winner is the argmin-cost candidate, and
eligibility is cost strictly below CostModel::canonical_cost. Strict <
comparisons preserve registration-order tie-breaking exactly as before.
The legacy comparators whose logic moved into SizeCost and the selection
loop (is_better_score, EstimateScore::{is_valid, beats}) are deleted.
CascadingCompressor gains with_cost_model(Arc<dyn CostModel>), holding
SizeCost by default, so default behavior is unchanged (bit-exact under
SizeCost by decision 1 of the cost-model plan; golden suite shows zero
snapshot churn). AlwaysUse short-circuiting, the deferred-callback ratio
threshold, and the byte-acceptance gate are untouched.
The winner_compress trace span gains estimated_cost alongside
estimated_ratio, keeping estimator observability at parity for
non-ratio models. WinnerEstimate::Score becomes a struct variant
carrying the winning cost; the choose_best_scheme unit tests' match
patterns are updated mechanically for the new variant shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
CostModel gains lower_bound(scheme, data) -> Option<Cost>: a bound on the best cost any candidate from the scheme could achieve. Pass 2 of choose_best_scheme now skips a scheme's deferred sampling or callback outright when the bound is not strictly below the best cost so far. This is the hook that later lets decode-heavy models skip expensive FSST sampling when it cannot win. The default implementation returns None (never prune), which SizeCost keeps, so behavior is unchanged (zero golden snapshot churn). The plan sketched the signature as returning Cost with a never-prunes default, but Cost must be finite and its units are model-defined, so no sentinel 'never prunes' value exists; Option<Cost> expresses the default honestly. Unit tests cover pruning behind a best candidate and the no-best case where pruning must not fire. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
BREAKING (changelog/break): the EstimateFn callback signature changes — the Option<EstimateScore> best-so-far parameter becomes a SkipThreshold handle owned by the compressor. Migration is mechanical: replace best_so_far.and_then(EstimateScore::finite_ratio) + 'max_ratio <= t' with threshold.best_case_ratio_cannot_win(max_ratio); the handle also exposes the old ratio view via best_ratio(). The handle wraps the best cost so far, the compressor's cost model, and the selection-site facts. best_case_ratio_cannot_win prices the calling scheme's hypothetical best-case candidate under the model and compares it against the best cost, so scheme-side early exits now speak cost rather than hardcoding ratio comparisons. Under the default SizeCost it reduces to exactly the historical 'max_ratio <= best_ratio' skip (pinned by unit tests across tie and one-ulp boundaries, and through the compressor-built handle), so behavior is unchanged: zero golden snapshot churn across all three variants. Delta and Sequence — the two callback users — migrate to the handle with identical max-ratio expressions (full_width * DELTA_PENALTY and len / 2), and gain end-to-end selection tests on both sides of their thresholds: winning over a low incumbent and losing at the exact best-case tie, plus Delta's min_ratio skip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
cost-model-r3-lower-bound
branch
from
f4617b8 to
d7a3b3a
Compare
connortsui20
force-pushed
the
cost-model-r2-costmodel
branch
2 times, most recently
from
7672d77 to
cf51915
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking Issue: #8701
Rationale for this change
Scheme estimation callbacks early-exit by comparing their best-case ratio against the best ratio seen so far — which hardcodes the size objective into every scheme. This PR generalizes the early-exit to the compressor's cost model so it stays correct under any model (#7697), and adds a model-side hook to prune deferred estimation work (e.g. expensive FSST sampling) that cannot win.
Stacked on #8698. Behavior is unchanged under the default
SizeCost: zero golden snapshot churn.What changes are included in this PR?
CostModel::lower_bound(default: never prune): selection skips a scheme's deferred sampling/callback outright when even its best case cannot beat the running best.EstimateFn'sOptionparameter becomes aSkipThresholdhandle: schemes askbest_case_ratio_cannot_win(max_ratio)and the handle prices the best case under the model. UnderSizeCostthis reduces to exactly the oldmax_ratio <= best_ratioskip — pinned by tests at exact-tie and one-ulp boundaries. Delta and Sequence (the two callback users) migrate and gain selection tests on both sides of their thresholds.What APIs are changed? Are there any user-facing changes?
Breaking (
changelog/break): theEstimateFnsignature changes; third-party schemes usingDeferredEstimate::Callbackmust migrate. The migration is mechanical — replace the ratio comparison withthreshold.best_case_ratio_cannot_win(max_ratio); the old ratio view remains available viabest_ratio().CostModel::lower_boundis additive (defaulted). Default compressor output is byte-identical.