Skip to content

Remove columns from iceberg export #249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gbrgr merged 5 commits into from
Apr 15, 2026
Merged

Remove columns from iceberg export #249

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6fbb2c7
Remove columns from iceberg export
gbrgr Apr 15, 2026
c005bb4
Update artifacts
gbrgr Apr 15, 2026
182742f
Remove export column from equality
gbrgr Apr 15, 2026
73781ba
Add reserved field
gbrgr Apr 15, 2026
f7bce26
Scope buf.yaml
gbrgr Apr 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion buf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ modules:
use:
- FILE
except:
- FIELD_NO_DELETE
- FIELD_NO_DELETE
- MESSAGE_NO_DELETE

@comnik comnik Apr 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm these should not be generic exceptions, I'm not sure how that slipped in. We should have to explicitly override the compatibility check when we remove messages or delete fields. In cases like this one it is ok to do so.

@gbrgr gbrgr Apr 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I scoped it down to the file. Is that OK?

@comnik comnik Apr 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now it is, I took a note to look into this more.

25 changes: 4 additions & 21 deletions meta/src/meta/grammar.y
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,6 @@
%nonterm export_csv_config transactions.ExportCSVConfig
%nonterm export_csv_path String
%nonterm export_csv_source transactions.ExportCSVSource
%nonterm export_iceberg_column transactions.ExportColumn
%nonterm export_iceberg_columns Sequence[transactions.ExportColumn]
%nonterm export_iceberg_config transactions.ExportIcebergConfig
%nonterm export_iceberg_table_def logic.RelationId
%nonterm iceberg_auth_properties Sequence[Tuple[String, String]]
Expand Down Expand Up @@ -1222,13 +1220,6 @@ iceberg_catalog_config
$5: Sequence[Tuple[String, String]] = builtin.dict_to_pairs($$.properties)
$6: Sequence[Tuple[String, String]] = builtin.dict_to_pairs($$.auth_properties)

export_iceberg_column
: "(" "column" STRING boolean_value ")"
construct: $$ = transactions.ExportColumn(name=$3, nullable=$4)
deconstruct:
$3: String = $$.name
$4: Boolean = $$.nullable

iceberg_to_snapshot
: "(" "to_snapshot" STRING ")"
construct: $$ = $3
Expand Down Expand Up @@ -1372,26 +1363,20 @@ export_iceberg_table_def
construct: $$ = $3
deconstruct: $3: logic.RelationId = $$

export_iceberg_columns
: "(" "columns" export_iceberg_column* ")"
construct: $$ = $3
deconstruct: $3: Sequence[transactions.ExportColumn] = $$

iceberg_table_properties
: "(" "table_properties" iceberg_property_entry* ")"
construct: $$ = $3
deconstruct: $3: Sequence[Tuple[String, String]] = $$

export_iceberg_config
: "(" "export_iceberg_config" iceberg_locator iceberg_catalog_config export_iceberg_table_def export_iceberg_columns iceberg_table_properties config_dict? ")"
construct: $$ = construct_export_iceberg_config_full($3, $4, $5, $6, $7, $8)
: "(" "export_iceberg_config" iceberg_locator iceberg_catalog_config export_iceberg_table_def iceberg_table_properties config_dict? ")"
construct: $$ = construct_export_iceberg_config_full($3, $4, $5, $6, $7)
deconstruct:
$3: logic.IcebergLocator = $$.locator
$4: logic.IcebergCatalogConfig = $$.config
$5: logic.RelationId = $$.table_def
$6: Sequence[transactions.ExportColumn] = $$.columns
$7: Sequence[Tuple[String, String]] = builtin.dict_to_pairs($$.table_properties)
$8: Optional[Sequence[Tuple[String, logic.Value]]] = deconstruct_export_iceberg_config_optional($$)
$6: Sequence[Tuple[String, String]] = builtin.dict_to_pairs($$.table_properties)
$7: Optional[Sequence[Tuple[String, logic.Value]]] = deconstruct_export_iceberg_config_optional($$)


%%
Expand Down Expand Up @@ -1770,7 +1755,6 @@ def construct_export_iceberg_config_full(
locator: logic.IcebergLocator,
config: logic.IcebergCatalogConfig,
table_def: logic.RelationId,
columns: Sequence[transactions.ExportColumn],
table_property_pairs: Sequence[Tuple[String, String]],
config_dict: Optional[Sequence[Tuple[String, logic.Value]]],
) -> transactions.ExportIcebergConfig:
Expand All @@ -1783,7 +1767,6 @@ def construct_export_iceberg_config_full(
locator=locator,
config=config,
table_def=table_def,
columns=columns,
prefix=builtin.some(prefix),
target_file_size_bytes=builtin.some(target_file_size_bytes),
compression=compression,
Expand Down
6 changes: 0 additions & 6 deletions proto/relationalai/lqp/v1/transactions.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,16 +122,10 @@ message ExportCSVSource {

// Iceberg Export config

message ExportColumn {
string name = 1;
bool nullable = 2;
}

message ExportIcebergConfig {
IcebergLocator locator = 1;
IcebergCatalogConfig config = 2;
RelationId table_def = 3; // Which definition to export as a table.
repeated ExportColumn columns = 4;
Comment thread
gbrgr marked this conversation as resolved.
optional string prefix = 5; // File name prefix for parquet files that are produced.
optional int64 target_file_size_bytes = 6;
string compression = 7;
Expand Down
Loading
Loading