feat(vortex-file): gate the file writer behind editions#8877
feat(vortex-file): gate the file writer behind editions#8877joseph-isaacs wants to merge 12 commits into
Conversation
bf7ca36 to
2da25df
Compare
Polar Signals Profiling ResultsLatest Run
Previous Runs (5)
Powered by Polar Signals Cloud |
Benchmarks: Vortex queries 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.007x ➖, 0↑ 0↓)
datafusion / parquet (0.945x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.936x ➖, 0↑ 0↓)
duckdb / parquet (0.992x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
…at layout to allow composing encoding validation with different layout strategies This will be used to adding edition encoding filtering Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
c84896c to
58776e3
Compare
…ions-gating-8qy1fv Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> # Conflicts: # vortex-file/src/strategy.rs
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.145x ❌ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.145x ❌, 0↑ 7↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.039x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.051x ➖, 0↑ 0↓)
datafusion / parquet (1.028x ➖, 1↑ 1↓)
datafusion / arrow (1.091x ➖, 0↑ 10↓)
duckdb / vortex-file-compressed (1.056x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.060x ➖, 0↑ 3↓)
duckdb / parquet (1.001x ➖, 2↑ 0↓)
duckdb / duckdb (1.060x ➖, 0↑ 2↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.007x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.011x ➖, 0↑ 0↓)
datafusion / parquet (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.994x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.003x ➖, 0↑ 0↓)
duckdb / parquet (1.005x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.018x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.015x ➖, 0↑ 0↓)
datafusion / parquet (1.016x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed (1.019x ➖, 1↑ 4↓)
duckdb / vortex-compact (1.015x ➖, 0↑ 2↓)
duckdb / parquet (1.011x ➖, 1↑ 1↓)
duckdb / duckdb (1.027x ➖, 0↑ 4↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.043x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.084x ➖, 0↑ 0↓)
datafusion / parquet (1.012x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.089x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.080x ➖, 0↑ 0↓)
duckdb / parquet (1.013x ➖, 0↑ 0↓)
|
Benchmarks: Statistical and Population Genetics 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (1.020x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.005x ➖, 0↑ 0↓)
duckdb / parquet (1.020x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.991x ➖, 1↑ 1↓)
datafusion / parquet (1.027x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.125x ❌, 0↑ 4↓)
duckdb / parquet (1.023x ➖, 0↑ 0↓)
duckdb / duckdb (1.020x ➖, 0↑ 0↓)
File Size Changes (201 files changed, +0.0% overall, 105↑ 96↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.982x ➖, 4↑ 0↓)
datafusion / vortex-compact (0.978x ➖, 3↑ 0↓)
datafusion / parquet (1.007x ➖, 1↑ 0↓)
datafusion / arrow (1.000x ➖, 2↑ 1↓)
duckdb / vortex-file-compressed (0.996x ➖, 2↑ 0↓)
duckdb / vortex-compact (1.020x ➖, 0↑ 0↓)
duckdb / parquet (0.998x ➖, 0↑ 0↓)
duckdb / duckdb (0.961x ➖, 4↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.930x ➖, 7↑ 0↓)
datafusion / parquet (0.942x ➖, 7↑ 1↓)
duckdb / vortex-file-compressed (0.966x ➖, 1↑ 2↓)
duckdb / parquet (0.981x ➖, 0↑ 0↓)
duckdb / duckdb (0.963x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.086x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.101x ➖, 0↑ 5↓)
datafusion / parquet (0.967x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (1.048x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.081x ➖, 0↑ 0↓)
duckdb / parquet (1.030x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Vortex (geomean): 1.069x ➖ How to read Verdict and Engines
unknown / unknown (1.054x ➖, 0↑ 7↓)
|
Benchmarks: Appian on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.925x ➖, 0↑ 0↓)
datafusion / parquet (0.913x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed (0.924x ➖, 1↑ 0↓)
duckdb / parquet (0.932x ➖, 1↑ 0↓)
duckdb / duckdb (0.933x ➖, 1↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=10 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.988x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.994x ➖, 0↑ 1↓)
datafusion / parquet (0.967x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (0.988x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.011x ➖, 0↑ 0↓)
duckdb / parquet (1.032x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Vortex (geomean): 1.009x ➖ How to read Verdict and Engines
unknown / unknown (1.013x ➖, 0↑ 8↓)
|
c85b1c3 to
a388e2e
Compare
…ions-gating-8qy1fv Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> # Conflicts: # vortex-file/src/writer.rs
Merging this PR will degrade performance by 0.17%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
…nks, Python goldens The develop merge brought in the file-level metadata tests (#8740), which write with bare sessions that enable no editions and now fail validation. Enable the crate's test edition in those sessions, matching the other vortex-file unit tests. Also re-qualify the vortex::editions module-doc links (module docs resolve at the lib.rs `pub mod` site) and update the Python doctest golden file sizes for the edition-derived footer context. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2
The writer now rejects encodings outside the session's enabled editions regardless of the configured strategy, so vortex-jni's strategy-level allow-set override for `vortex.parquet.variant` no longer worked. JNI sessions instead enable the latest unstable edition (the family that encoding belongs to), which restores variant writability through the session policy. The schema-conditional strategy override is removed: parquet-variant arrays only arise for variant-typed columns, so the session-wide policy is equivalent. Sibling unstable encodings are not registered in the JNI build, so the footer's pre-populated context may advertise enabled-but-unregistered ids; a new integration test pins that such files stay readable. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2
c5aff5c to
a5e0c3d
Compare
Rationale for this change
Follow-up to the editions implementation (#8871): the edition declarations existed but nothing enforced them. This PR gates the file writer behind editions. The session is the source of truth: it stores the set of enabled editions alongside the declarations, and exposes the resolved set of all enabled encodings. The default session enables
core2026.07.0(EditionId::new("core", 2026, 7, 0)), so by default:What changes are included in this PR?
vortex-edition:EditionSessionnow stores the enabled editions next to the declarations:enable()(must be declared; at most one edition per family),enabled(), andenabled_encodings()— the resolved, sorted union of the enabled editions' encoding sets (Nonewhen nothing is enabled). The first-party declarations stay in thevortexfacade; this crate remains types + session variable + test harness.vortex(facade):register_default_editionsnow also enablesDEFAULT_WRITE_EDITIONS(core2026.07.0, plus the newestunstabledraft when theunstable_encodingsfeature opts in) on the session. New tests: a golden pin of the frozencore2026.07.0encoding set, per-edition harness validation, subset properties, and a check that every default compression scheme's produced encodings are covered by the default write editions.vortex-compressor: new requiredScheme::produced_encodings(&self) -> Vec<ArrayId>trait method declaring the array encodings a scheme may emit (empty = only canonical arrays; cascaded children are covered by their own schemes).CascadingCompressor::with_enabled_editions_gatingmakes scheme selection resolve the session's enabled encodings from the execution context at compression time and skip schemes whose outputs are not all allowed; sessions with no enabled editions are not gated. Unit-tested with a synthetic edition/scheme.vortex-btrblocks: every scheme declares its produced encodings (including the env-gatedPatchedoutput of BitPacking/ALP).BtrBlocksCompressorBuildergainsgate_by_enabled_editions()(session gating pass-through) andretain_allowed_encodings()(static filtering for explicit allow sets).vortex-layout:FlatLayoutStrategy::with_enabled_editions_gatingresolves the allowed encodings from the session's enabled editions atwrite_streamtime — an encoding outside them fails the write via the existing normalize-with-Errorvalidation — falling back to the explicitallowed_encodingsset when the session enables no editions.vortex-file: the defaultWriteStrategyBuilderwires all of this up: flat writer gated by enabled editions with the staticALLOWED_ENCODINGSregistry as fallback, and both the data and stats compressors gated at compression time. An explicitwith_allow_encodingsset overrides the session gate everywhere (unchanged behavior forvortex-jni/vortex-parquet-variant). The writer's deterministic pre-populatedArrayContextlikewise comes from the session's enabled encodings. Experimental patches remain a runtime opt-out:Patchedis only writable when the experimental environment variable is set, mirroring when compression produces it.Checks run (after rebasing onto
developat #8871):cargo nextest run -p vortex-edition -p vortex-compressor -p vortex-btrblocks -p vortex-layout -p vortex-file -p vortex(411 passed; also with--all-featuresfor file/btrblocks/compressor),cargo test --docfor the touched crates,cargo +nightly fmt --all, andcargo clippy --all-targets --all-featuresfor the touched crates plusvortex-tensor,vortex-jni, andvortex-parquet-variant(clean). Workspace-wide clippy could not be run:vortex-duckdb's build script cannot download DuckDB sources in this sandbox.What APIs are changed? Are there any user-facing changes?
Schemehas a new required methodproduced_encodings() -> Vec<ArrayId>— a breaking change for third-party schemes, which must now declare their outputs.EditionSession::{enable, enabled, enabled_encodings},CascadingCompressor::with_enabled_editions_gating,BtrBlocksCompressorBuilder::{gate_by_enabled_editions, retain_allowed_encodings},FlatLayoutStrategy::with_enabled_editions_gating, andvortex::editions::DEFAULT_WRITE_EDITIONSare new.vortexfacade: encodings outside the enabled editions (fastlanes.delta,vortex.onpair,vortex.zstd_bufferswithout theunstable_encodingsfeature,vortex.patchedwithout the env opt-in) are no longer writable — writes containing them fail at the flat-layout validation instead of producing files without the read-forever guarantee. Sessions that enable no editions keep the previous staticALLOWED_ENCODINGSbehavior.🤖 Generated with Claude Code
https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2