Skip to content

Relocate Delta's selection policy into cost-model priors; plumb the model through the builders#8700

Draft
connortsui20 wants to merge 5 commits into
cost-model-r3-lower-boundfrom
cost-model-r5-builders
Draft

Relocate Delta's selection policy into cost-model priors; plumb the model through the builders#8700
connortsui20 wants to merge 5 commits into
cost-model-r3-lower-boundfrom
cost-model-r5-builders

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 9, 2026

Copy link
Copy Markdown
Member

Tracking Issue: #8701

Rationale for this change

Delta's selection policy — the 0.95 "delta tax" and the 1.25 minimum-win floor — is an execution-speed judgment expressed as ratio fudge factors inside the scheme. This PR moves that policy onto the cost model as data (a per-scheme prior), and makes the model configurable end to end: through BtrBlocksCompressorBuilder and, via the file writer's strategy, into both the data and stats compressors (#7697).

Stacked on #8699. Behavior-preserving by construction: the model applies the identical arithmetic in the identical order, so scores are bit-identical — zero golden snapshot churn, including the variant where Delta wins corpus entries.

What changes are included in this PR?

  • SizeCost gains per-scheme priors (SchemePrior { multiplier, min_ratio }); DeltaScheme reports raw ratios, and its prior is registered on the btrblocks default cost model, where the policy is documented as policy, not measurement.
  • BtrBlocksCompressorBuilder::with_cost_model; a configured model rides through WriteStrategyBuilder into both writer compressors. Liveness tests prove the plumbing drives selection (an inverted model flips the winner, and writes a strictly larger file).

What APIs are changed? Are there any user-facing changes?

Additive: SchemePrior, SizeCost::with_scheme_prior, BtrBlocksCompressorBuilder::with_cost_model, and a cost re-export from vortex-btrblocks. Deprecated: DeltaScheme::new(min_ratio) — the floor belongs on the model; the constructor keeps working and can only tighten policy. Embedders assembling CascadingCompressor directly with DeltaScheme should attach a prior-bearing model (as the builder does) to retain the handicap. Default compressor output is byte-identical.

@connortsui20 connortsui20 added the changelog/deprecation A change that introduces a series of API deprecations label Jul 9, 2026 — with Claude
@connortsui20 connortsui20 marked this pull request as draft July 9, 2026 15:42
@connortsui20 connortsui20 force-pushed the cost-model-r5-builders branch from cfa1afe to a380163 Compare July 9, 2026 17:17
@connortsui20 connortsui20 force-pushed the cost-model-r3-lower-bound branch from 56f8b12 to f4617b8 Compare July 9, 2026 17:17
claude added 5 commits July 10, 2026 09:10
SizeCost gains a prior table keyed by SchemeId: SchemePrior carries a
multiplier applied to the scheme's raw ratio signal (values below 1.0
demand the scheme win by a margin) and a floor the effective ratio must
strictly exceed. Priors apply in candidate pricing and, through it, in
the SkipThreshold best-case path, so scheme-side and model-side views
of a handicapped scheme stay consistent. Schemes without an entry are
priced off their raw ratio exactly as before, and SizeCost::default()
registers no priors, so behavior is unchanged.

Priors are policy, not measurement: this is the home for judgments like
the Delta 'tax', which moves here from the scheme in the next commit.
SizeCost gives up Copy for the table; it remains Clone + Default.

Unit tests pin the prior arithmetic bit-for-bit against the historical
scheme-side expression (raw * multiplier, floor comparison included)
in both the pricing and threshold paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
DeltaScheme now reports its raw estimated ratio (full_width /
delta_bits) and asks the threshold about its raw best case; the 'delta
tax' multiplier (0.95) and minimum-win floor (1.25) are registered as
Delta's SchemePrior on the btrblocks default cost model, which
BtrBlocksCompressorBuilder::build() now attaches. The model applies the
identical arithmetic in the identical order (raw * multiplier, floor
comparison), so scores and selections are bit-identical: the golden
suite — including the unstable variant where Delta wins
int_monotone_jitter — shows zero snapshot churn.

The ALL_SCHEMES 'prefer above delta' registration comment moves to
default_cost_model, where the policy now lives and is documented as
policy, not measurement.

DeltaScheme::new(min_ratio) is deprecated: the floor belongs on the
model (SizeCost::with_scheme_prior). The constructor keeps working —
a configured floor still gates via the historical post-penalty
comparison — but it can only tighten policy, since the model's prior
floor also applies. Note for embedders: assembling CascadingCompressor
directly with DeltaScheme now requires attaching a prior-bearing model
(as the builder does) to retain the handicap; the scheme alone no
longer penalizes itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The builder now carries the cost model and attaches it to the
CascadingCompressor at build(), defaulting to the btrblocks default
model (SizeCost with Delta's prior) as before, so default behavior is
unchanged (zero golden snapshot churn). The vortex_compressor::cost
module is re-exported from vortex-btrblocks alongside the existing
framework re-exports so builder users can name the model types.

A unit test proves the plumbing is live rather than inert: two
fixed-ratio marker schemes are selected differently under the default
model and under an inverted model injected through the builder.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
CompressorConfig::BtrBlocks holds the whole BtrBlocksCompressorBuilder,
so a cost model configured via with_cost_model now rides through
WriteStrategyBuilder::build() into both the data compressor (which
clones the builder and excludes IntDictScheme) and the stats/values
compressor, with no strategy code changes needed; this change documents
that on with_btrblocks_builder and proves it with an integration test:
on the same highly compressible column, a strategy carrying an inverted
model writes a strictly larger file than the default strategy.

No Python surface is added yet, deviating deliberately from the
NEXT-STEPS strawman ('Python preset stubs'): a Python surface with only
the default model behind it is confusable dead API; it lands with the
first real preset in C1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
DeltaScheme's public docs linked to the private builder module,
tripping rustdoc's private_intra_doc_links warning; refer to the
default cost model in prose instead. (Belongs with the R4 delta-prior
change; kept as a follow-up to avoid rewriting pushed history.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 force-pushed the cost-model-r3-lower-bound branch from f4617b8 to d7a3b3a Compare July 10, 2026 09:18
@connortsui20 connortsui20 force-pushed the cost-model-r5-builders branch from a380163 to b81955f Compare July 10, 2026 09:18
@codspeed-hq

codspeed-hq Bot commented Jul 10, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.86%

⚡ 11 improved benchmarks
❌ 1 regressed benchmark
✅ 1607 untouched benchmarks
🆕 11 new benchmarks
⏩ 42 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_into_canonical[(1000, 10)] 169.6 µs 205.8 µs -17.57%
Simulation rebuild_naive 109.1 µs 91.1 µs +19.76%
Simulation pushdown_compare[(1000, 16, 4)] 150.1 µs 128 µs +17.2%
Simulation pushdown_compare[(1000, 64, 4)] 149.9 µs 128.2 µs +16.92%
Simulation pushdown_compare[(1000, 4, 4)] 149.6 µs 128.2 µs +16.72%
Simulation pushdown_compare[(1000, 64, 8)] 156.2 µs 134.3 µs +16.26%
Simulation pushdown_compare[(1000, 4, 8)] 154 µs 133 µs +15.76%
Simulation pushdown_compare[(1000, 16, 8)] 162 µs 140.5 µs +15.34%
Simulation chunked_varbinview_into_canonical[(100, 100)] 307 µs 271.6 µs +13.02%
Simulation encode_varbin[(1000, 32)] 165 µs 147.4 µs +11.97%
Simulation chunked_varbinview_opt_canonical_into[(100, 100)] 340.5 µs 305.3 µs +11.54%
Simulation compare_int_constant 298 µs 268.7 µs +10.89%
🆕 Simulation fsl_sum_large N/A 65.2 ms N/A
🆕 Simulation fsl_sum_medium N/A 727.9 µs N/A
🆕 Simulation fsl_sum_small N/A 56.8 µs N/A
🆕 Simulation list_sum_large N/A 135.7 ms N/A
🆕 Simulation list_sum_medium N/A 1.1 ms N/A
🆕 Simulation list_sum_nullable_elements_large N/A 495.9 ms N/A
🆕 Simulation list_sum_nullable_elements_medium N/A 4.9 ms N/A
🆕 Simulation list_sum_small N/A 144.2 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cost-model-r5-builders (b81955f) with cost-model-r3-lower-bound (f4617b8)2

Open in CodSpeed

Footnotes

  1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on cost-model-r3-lower-bound (d7a3b3a) during the generation of this report, so 0c7981b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/deprecation A change that introduces a series of API deprecations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants