feat(encryption): parquet encryption via delta.encryption.* table properties

[corwinjoy]

Summary

Implements Parquet encryption driven by standard delta.encryption.* table properties stored in the delta log, following the Modular Encryption in Delta Lake RFC.

Key difference from previous approach: encryption is configured as table properties at creation time, not as a runtime API parameter. Once set, all operations (write, read, delete, update, merge, optimize) encrypt/decrypt automatically.

User-facing API

// 1. Register KMS factory once at startup
register_encryption_factory("my-kms", Arc::new(MyKmsFactory::new()));

// 2. Create encrypted table
table.create()
    .with_property("delta.encryption.kms.id",     "my-kms")
    .with_property("delta.encryption.footer.key",  "master-key-id")
    .with_property("delta.encryption.column.keys", "col-key:ssn,dob")
    .await?;

// 3. All subsequent operations are automatically encrypted/decrypted
table.write(batches).await?;   // encrypted
table.delete()...await?;       // decrypted read + encrypted write
table.scan_table().execute()?; // decrypted

Design

• EncryptionConfig parsed from TableProperties.unknown_properties follows the same from_config(table_config) pattern as WriterStatsConfig
• WriterPropertiesFactory async trait enables path-based KMS key derivation (AAD) — the file path is passed to create_writer_properties() before the parquet writer is created
• Global EncryptionFactoryRegistry: delta-rs operations create their own internal DataFusion sessions; users call register_encryption_factory(id, factory) once and the global registry bridges the gap
• DeltaScanConfig.table_parquet_options derived from snapshot table properties — reads decrypt via the registered factory
• Compact and Z-order optimize use DataFusion scan when encryption is configured

Files changed

• crates/core/src/table/config.rs — EncryptionConfig struct + EncryptionExt trait
• crates/core/src/operations/write/encryption.rs — NEW: WriterPropertiesFactory, WriterEncryptionConfig, global registry
• crates/core/src/operations/write/writer.rs — factory-based WriterConfig/LazyArrowWriter
• crates/core/src/operations/write/execution.rs — derives factory from table config
• crates/core/src/delta_datafusion/table_provider.rs — DeltaScanConfig.table_parquet_options
• crates/core/src/delta_datafusion/table_provider/next/scan/mod.rs — factory lookup in read path
• crates/core/src/operations/optimize.rs — encrypted compact/z-order reads
• crates/core/src/writer/json.rs + record_batch.rs — factory + path-based for AAD
• crates/core/src/test_utils/kms_encryption.rs — NEW: MockKmsFactory for testing
• crates/core/tests/commands_with_encryption.rs — NEW: integration tests
• crates/deltalake/examples/basic_operations_encryption.rs — NEW: end-to-end example

Test plan

• cargo build -p deltalake-core --features datafusion — clean
• cargo test -p deltalake-core --features datafusion — all pass
• cargo run --example basic_operations_encryption --features "datafusion integration-test" -p deltalake — end-to-end: create, write, Z-order, compact, read with KMS encryption

🤖 Generated with Claude Code

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[Copilot]

Pull request overview

This PR introduces Parquet encryption support in delta-rs driven by standard delta.encryption.* Delta table properties, including a process-wide registry to make KMS factories available to internally-created DataFusion sessions.

Changes:

• Adds EncryptionConfig parsing from table properties and propagates it into DataFusion scan options for decryption.
• Introduces an async, path-aware WriterPropertiesFactory abstraction plus a global encryption factory registry for write-time key retrieval.
• Updates write + optimize execution paths (and adds tests/example) to use encryption-aware readers/writers when encryption is configured.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
crates/deltalake/examples/basic_operations_encryption.rs	New end-to-end example showing encrypted create/write/optimize/read using table properties.
crates/deltalake/Cargo.toml	Adds an integration-test feature and wires the new example behind feature gates.
crates/core/tests/commands_with_encryption.rs	New integration tests covering encrypted create/read/optimize/delete/update.
crates/core/src/writer/record_batch.rs	Switches RecordBatchWriter to a per-file WriterProperties factory and AAD path precomputation.
crates/core/src/writer/json.rs	Switches JsonWriter to a per-file WriterProperties factory and AAD path precomputation.
crates/core/src/test_utils/mod.rs	Exposes kms_encryption test utils when datafusion feature is enabled.
crates/core/src/test_utils/kms_encryption.rs	Adds a mock KMS EncryptionFactory for tests/examples.
crates/core/src/table/config.rs	Adds EncryptionConfig parsing and conversion to DataFusion TableParquetOptions.
crates/core/src/operations/write/writer.rs	Refactors writer config to use async WriterPropertiesFactory (supports KMS + AAD).
crates/core/src/operations/write/mod.rs	Exposes new operations::write::encryption module.
crates/core/src/operations/write/execution.rs	Resolves writer factory from explicit props vs table encryption vs session defaults.
crates/core/src/operations/write/encryption.rs	New writer-side encryption module: factory abstraction + global registry.
crates/core/src/operations/optimize.rs	Uses encryption-aware read path and writer factory for compact/z-order when configured.
crates/core/src/delta_datafusion/table_provider.rs	Propagates table encryption options into scan config and scan-time decryption factory wiring.
crates/core/src/delta_datafusion/table_provider/next/scan/mod.rs	Passes table parquet options through to scans and wires decryption factory lookup on read.
crates/core/src/delta_datafusion/mod.rs	Re-exports FileSelection for optimize compact encrypted path.
crates/core/Cargo.toml	Enables Parquet encryption feature; adds dev-dependency.
Cargo.toml	Enables DataFusion parquet_encryption feature at workspace level.

Comments suppressed due to low confidence (2)

crates/core/src/table/config.rs:597

• EncryptionConfig::to_table_parquet_options() starts from TableParquetOptions::default() and only sets crypto fields. Call sites then use this value as a full replacement for the session’s parquet options, which can unintentionally reset non-crypto parquet settings (pushdown, schema options, etc.) for encrypted tables. Consider storing only the crypto overrides and merging them into the session’s TableParquetOptions at scan-build time, or populate opts.global from the session config when constructing scan options.

    pub fn to_table_parquet_options(&self) -> TableParquetOptions {
        let mut opts = TableParquetOptions::default();
        opts.crypto.factory_id = Some(self.kms_id.clone());
        opts.crypto.factory_options = self.factory_options();
        opts
    }

crates/core/src/table/config.rs:610

• EncryptionConfig::factory_options() currently forwards only kms.configuration and footer.key. The parsed plaintext_footer and column_keys properties are never forwarded to the factory, so those table properties have no effect. Either encode these into EncryptionFactoryOptions (with stable option keys) or remove them from EncryptionConfig until supported to avoid misleading API behavior.

    /// Build [`EncryptionFactoryOptions`] from the KMS configuration string.
    #[cfg(feature = "datafusion")]
    pub fn factory_options(&self) -> EncryptionFactoryOptions {
        let mut opts = EncryptionFactoryOptions::default();
        if let Some(cfg) = &self.kms_configuration {
            opts.options
                .insert("kms.configuration".to_string(), cfg.clone());
        }
        opts.options
            .insert("footer.key".to_string(), self.footer_key.clone());
        opts
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 7 comments.