[#11072] feat(trino-connector): Add Glue catalog connector adapter#11073
Open
diqiu50 wants to merge 8 commits into
Open
[#11072] feat(trino-connector): Add Glue catalog connector adapter#11073diqiu50 wants to merge 8 commits into
diqiu50 wants to merge 8 commits into
Conversation
- Upgrade AWS SDK from 2.29.52 to 2.31.73 to access IcebergTableUpdate and related Glue native Iceberg alter APIs - createTable: route tables with table-format=iceberg through OpenTableFormatInput.IcebergInput.CREATE so Glue writes metadata.json - alterTable: detect existing Iceberg tables via table_type=ICEBERG and route through UpdateTable + UpdateOpenTableFormatInput + IcebergTableUpdate, keeping Iceberg metadata consistent instead of only updating Glue columns - Add GlueIcebergHelper with type conversion (Hive/Gravitino → Iceberg Document), field-ID-preserving schema building, and column change application - Fix matchesFormatFilter to recognize Iceberg tables created via default-table-format config (table_type=ICEBERG without table-format key) - Add 30 unit tests covering routing, type conversion, schema evolution, field ID tracking, and error paths
Code Coverage Report
Files
|
…rms and fix GlueMetadataAdapter properties - Use Iceberg SDK to create Iceberg tables in Glue (writes metadata.json to S3) - Recover Iceberg partitioning and sort orders from Iceberg metadata on loadTable - Add GlueIcebergCatalogHelper for Iceberg table creation via GlueCatalog - Refactor GlueMetadataAdapter to use ExpressionUtil for partition expression parsing - Fix bucket_count to integerProperty and sorted_by to ArrayType property metadata - Add TestGlueMetadataAdapter for round-trip partition/bucket/sort verification
- Combine GlueIcebergHelper and GlueIcebergCatalogHelper into a single GlueIcebergTableHelper that delegates all Iceberg operations to the Iceberg SDK GlueCatalog. - createTable: uses Iceberg SDK for non-register-mode tables. - alterTable: uses Iceberg SDK UpdateSchema/UpdateProperties for SDK-managed tables; falls back to native Glue SDK for register-mode tables (METADATA_LOCATION present). - loadTable: recovers partitioning and sort orders from Iceberg metadata. - Update TestGlueIceberg to match the new delegation paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add Trino connector support for Gravitino's Glue catalog:
GlueConnectorAdapter– wires Gravitino Glue catalog properties into Trino's lakehouse connector configGlueMetadataAdapter– converts table/schema properties between Gravitino and Trino formatsGlueTablePropertyConverter– maps Glue-specific table properties (table-format → type, etc.)Also includes fixes to
GlueCatalogOperationsandGlueIcebergHelperfound during integration testing.Why are the changes needed?
Without a dedicated connector adapter, Trino cannot proxy queries to Glue-managed tables (Hive, Iceberg) through Gravitino.
Fix: #11072
Does this PR introduce any user-facing change?
Yes — users can now configure a Trino catalog backed by Gravitino's Glue catalog.
How was this patch tested?
Manual integration tests with Trino against a real AWS Glue environment.