Skip to content

Commit 70143ca

Browse files
ding-youngDrakeLin
andauthored
refactor: unify Reader/WriterFeature into a single TableFeature (delta-io#1345)
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? closes delta-io#1340 As suggested in above issue, this pr - adds validation check for given (reader_features, writer_features) on `Protocol::try_new` - refactors `ReaderFeature` `WriterFeature` into single `TableFeature` so that we can check only writer feature like ```rust // If protocol is created via Protocol::try_new(), then it is valid if !protocol.has_write_feature(&TableFeature::VariantType) ``` Note that there are three feature types, and the `Unknown` variant is for handling `TableFeature::Unknown` which is either `Writer` or `ReaderWriter` based on its actual usage. ```rust pub(crate) enum FeatureType { Writer, ReaderWriter, Unknown, } ``` <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Yes, added new test --------- Co-authored-by: Drake Lin <drakelin18@gmail.com>
1 parent 5058bce commit 70143ca

12 files changed

Lines changed: 406 additions & 348 deletions

File tree

kernel/src/actions/crc.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,7 @@ mod tests {
150150
use crate::engine::sync::SyncEngine;
151151
use crate::schema::derive_macro_utils::ToDataType as _;
152152
use crate::schema::{ArrayType, DataType, StructField, StructType};
153-
use crate::table_features::{ReaderFeature, WriterFeature};
153+
use crate::table_features::TableFeature;
154154
use crate::utils::test_utils::string_array_to_engine_data;
155155
use crate::Engine;
156156

@@ -252,8 +252,8 @@ mod tests {
252252
let expected_protocol = Protocol {
253253
min_reader_version: 3,
254254
min_writer_version: 7,
255-
reader_features: Some(vec![ReaderFeature::ColumnMapping]),
256-
writer_features: Some(vec![WriterFeature::ColumnMapping]),
255+
reader_features: Some(vec![TableFeature::ColumnMapping]),
256+
writer_features: Some(vec![TableFeature::ColumnMapping]),
257257
};
258258
let expected_metadata = Metadata {
259259
id: "testId".to_string(),

kernel/src/actions/mod.rs

Lines changed: 196 additions & 76 deletions
Large diffs are not rendered by default.

kernel/src/actions/visitors.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -686,7 +686,7 @@ mod tests {
686686

687687
use crate::engine::sync::SyncEngine;
688688
use crate::expressions::{column_expr_ref, Expression};
689-
use crate::table_features::{ReaderFeature, WriterFeature};
689+
use crate::table_features::TableFeature;
690690
use crate::utils::test_utils::{action_batch, parse_json_batch};
691691
use crate::Engine;
692692

@@ -697,8 +697,8 @@ mod tests {
697697
let expected = Protocol {
698698
min_reader_version: 3,
699699
min_writer_version: 7,
700-
reader_features: Some(vec![ReaderFeature::DeletionVectors]),
701-
writer_features: Some(vec![WriterFeature::DeletionVectors]),
700+
reader_features: Some(vec![TableFeature::DeletionVectors]),
701+
writer_features: Some(vec![TableFeature::DeletionVectors]),
702702
};
703703
assert_eq!(parsed, expected);
704704
Ok(())

kernel/src/engine/arrow_data.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -352,7 +352,7 @@ mod tests {
352352
use crate::engine::sync::SyncEngine;
353353
use crate::expressions::ArrayData;
354354
use crate::schema::{ArrayType, DataType, StructField, StructType};
355-
use crate::table_features::{ReaderFeature, WriterFeature};
355+
use crate::table_features::TableFeature;
356356
use crate::utils::test_utils::{assert_result_error_with_message, string_array_to_engine_data};
357357
use crate::{DeltaResult, Engine as _, EngineData as _};
358358

@@ -394,11 +394,11 @@ mod tests {
394394
assert_eq!(protocol.min_writer_version(), 7);
395395
assert_eq!(
396396
protocol.reader_features(),
397-
Some([ReaderFeature::unknown("rw1")].as_slice())
397+
Some([TableFeature::unknown("rw1")].as_slice())
398398
);
399399
assert_eq!(
400400
protocol.writer_features(),
401-
Some([WriterFeature::unknown("rw1"), WriterFeature::unknown("w2")].as_slice())
401+
Some([TableFeature::unknown("rw1"), TableFeature::unknown("w2")].as_slice())
402402
);
403403
Ok(())
404404
}

kernel/src/schema/variant_utils.rs

Lines changed: 15 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
33
use crate::actions::Protocol;
44
use crate::schema::{Schema, SchemaTransform, StructType};
5-
use crate::table_features::{ReaderFeature, WriterFeature};
5+
use crate::table_features::TableFeature;
66
use crate::utils::require;
77
use crate::{DeltaResult, Error};
88
use std::borrow::Cow;
@@ -24,10 +24,8 @@ pub(crate) fn validate_variant_type_feature_support(
2424
) -> DeltaResult<()> {
2525
// Both the reader and writer need to have either the VariantType or the VariantTypePreview
2626
// features.
27-
if (!protocol.has_reader_feature(&ReaderFeature::VariantType)
28-
&& !protocol.has_reader_feature(&ReaderFeature::VariantTypePreview))
29-
|| (!protocol.has_writer_feature(&WriterFeature::VariantType)
30-
&& !protocol.has_writer_feature(&WriterFeature::VariantTypePreview))
27+
if !protocol.has_table_feature(&TableFeature::VariantType)
28+
&& !protocol.has_table_feature(&TableFeature::VariantTypePreview)
3129
{
3230
let mut uses_variant = UsesVariant::default();
3331
let _ = uses_variant.transform_struct(schema);
@@ -46,7 +44,7 @@ mod tests {
4644
use super::*;
4745
use crate::actions::Protocol;
4846
use crate::schema::{DataType, StructField, StructType};
49-
use crate::table_features::{ReaderFeature, WriterFeature};
47+
use crate::table_features::TableFeature;
5048
use crate::utils::test_utils::assert_result_error_with_message;
5149

5250
#[test]
@@ -74,10 +72,10 @@ mod tests {
7472
#[test]
7573
fn test_variant_feature_validation() {
7674
let features = [
77-
(ReaderFeature::VariantType, WriterFeature::VariantType),
75+
(TableFeature::VariantType, TableFeature::VariantType),
7876
(
79-
ReaderFeature::VariantTypePreview,
80-
WriterFeature::VariantTypePreview,
77+
TableFeature::VariantTypePreview,
78+
TableFeature::VariantTypePreview,
8179
),
8280
];
8381
let schema_with_variant = StructType::new_unchecked([
@@ -120,15 +118,17 @@ mod tests {
120118
)
121119
.unwrap();
122120

123-
// Protocol without variantType writer feature
121+
// Since variant features are ReaderWriter feature, protocol that
122+
// lists a variant feature in only one of reader/writer feature = ERR
124123
let protocol_without_writer_feature =
125-
Protocol::try_new(3, 7, Some([variant_reader]), Some::<Vec<String>>(vec![]))
126-
.unwrap();
124+
Protocol::try_new(3, 7, Some([variant_reader]), Some::<Vec<String>>(vec![]));
125+
assert_result_error_with_message(protocol_without_writer_feature,
126+
"Reader features must contain only ReaderWriter features that are also listed in writer features");
127127

128-
// Protocol without variantType reader feature
129128
let protocol_without_reader_feature =
130-
Protocol::try_new(3, 7, Some::<Vec<String>>(vec![]), Some([variant_writer]))
131-
.unwrap();
129+
Protocol::try_new(3, 7, Some::<Vec<String>>(vec![]), Some([variant_writer]));
130+
assert_result_error_with_message(protocol_without_reader_feature,
131+
"Writer features must be Writer-only or also listed in reader features");
132132

133133
// Schema with VARIANT + Protocol with features = OK
134134
validate_variant_type_feature_support(
@@ -163,20 +163,6 @@ mod tests {
163163
&protocol_without_features,
164164
);
165165
assert_result_error_with_message(result, "Unsupported: Table contains VARIANT columns but does not have the required 'variantType' feature in reader and writer features");
166-
167-
// Schema with VARIANT + Protocol without writer feature = ERROR
168-
let result = validate_variant_type_feature_support(
169-
&schema_with_variant,
170-
&protocol_without_writer_feature,
171-
);
172-
assert_result_error_with_message(result, "Unsupported: Table contains VARIANT columns but does not have the required 'variantType' feature in reader and writer features");
173-
174-
// Schema with VARIANT + Protocol without reader feature = ERROR
175-
let result = validate_variant_type_feature_support(
176-
&schema_with_variant,
177-
&protocol_without_reader_feature,
178-
);
179-
assert_result_error_with_message(result, "Unsupported: Table contains VARIANT columns but does not have the required 'variantType' feature in reader and writer features");
180166
});
181167
}
182168
}

kernel/src/table_changes/log_replay/tests.rs

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ use crate::scan::state::DvInfo;
1010
use crate::scan::PhysicalPredicate;
1111
use crate::schema::{DataType, StructField, StructType};
1212
use crate::table_changes::log_replay::LogReplayScanner;
13-
use crate::table_features::ReaderFeature;
13+
use crate::table_features::TableFeature;
1414
use crate::utils::test_utils::{assert_result_error_with_message, Action, LocalMockTable};
1515
use crate::Predicate;
1616
use crate::{DeltaResult, Engine, Error, Version};
@@ -79,8 +79,8 @@ async fn metadata_protocol() {
7979
Protocol::try_new(
8080
3,
8181
7,
82-
Some([ReaderFeature::DeletionVectors]),
83-
Some([ReaderFeature::ColumnMapping]),
82+
Some([TableFeature::DeletionVectors]),
83+
Some([TableFeature::DeletionVectors]),
8484
)
8585
.unwrap(),
8686
),
@@ -138,8 +138,8 @@ async fn unsupported_reader_feature() {
138138
Protocol::try_new(
139139
3,
140140
7,
141-
Some([ReaderFeature::DeletionVectors, ReaderFeature::ColumnMapping]),
142-
Some([""; 0]),
141+
Some([TableFeature::DeletionVectors, TableFeature::ColumnMapping]),
142+
Some([TableFeature::DeletionVectors, TableFeature::ColumnMapping]),
143143
)
144144
.unwrap(),
145145
)])

kernel/src/table_changes/mod.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ use crate::log_segment::LogSegment;
4141
use crate::path::AsUrl;
4242
use crate::schema::{DataType, Schema, StructField, StructType};
4343
use crate::snapshot::{Snapshot, SnapshotRef};
44-
use crate::table_features::{ColumnMappingMode, ReaderFeature};
44+
use crate::table_features::{ColumnMappingMode, TableFeature};
4545
use crate::table_properties::TableProperties;
4646
use crate::utils::require;
4747
use crate::{DeltaResult, Engine, Error, Version};
@@ -272,8 +272,8 @@ fn check_cdf_table_properties(table_properties: &TableProperties) -> DeltaResult
272272
/// Ensures that Change Data Feed is supported for a table with this [`Protocol`] .
273273
/// See the documentation of [`TableChanges`] for more details.
274274
fn ensure_cdf_read_supported(protocol: &Protocol) -> DeltaResult<()> {
275-
static CDF_SUPPORTED_READER_FEATURES: LazyLock<Vec<ReaderFeature>> =
276-
LazyLock::new(|| vec![ReaderFeature::DeletionVectors]);
275+
static CDF_SUPPORTED_READER_FEATURES: LazyLock<Vec<TableFeature>> =
276+
LazyLock::new(|| vec![TableFeature::DeletionVectors]);
277277
match &protocol.reader_features() {
278278
// if min_reader_version = 3 and all reader features are subset of supported => OK
279279
Some(reader_features) if protocol.min_reader_version() == 3 => {

0 commit comments

Comments
 (0)