[IcebergIO] Upgrade Iceberg dependency to 1.11.0 - #39559
Merged
Merged
Conversation
Bumps iceberg-core/api/parquet/orc/data and the GCP/AWS/Azure runtime modules from 1.10.0 to 1.11.0. Iceberg 1.11.0 pulls parquet-avro/parquet-hadoop 1.17.1 transitively; pin them at 1.16.0 (current Beam parquet_version) so this PR stays zero-behavior-change. A separate PR can bump parquet_version once 1.17.x is vetted across the rest of Beam. The old per-format static builders (Parquet.read(), Avro.writeData(), ORC.write(), etc.) are only deprecated in 1.11.0, not removed, so RecordWriter / ScanTaskReader / ReadUtils still compile unchanged. A follow-up PR will migrate them to the new FormatModelRegistry SPI. Rebased on merged apache#39064 (Java 17 floor). Tracks apache#38925.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Abacn
reviewed
Jul 30, 2026
Contributor
|
Assigning reviewers: R: @kennknowles for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Abacn
reviewed
Jul 30, 2026
Co-authored-by: Yi Hu <huuyyi@gmail.com>
Abacn
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changes
Upgrades Apache Iceberg from 1.10.0 to 1.11.0, including its core, file-format, and cloud-storage components (iceberg-core/api/parquet/orc/data and the GCP/AWS/Azure runtime modules).
This PR is based on #39064, which raised IcebergIO’s Java requirement to Java 17. Tracks #38925.
What Does Not Change
Iceberg 1.11.0 would normally bring in newer Parquet libraries (1.17.1). This PR keeps Beam on its existing Parquet version (1.16.0) so the Iceberg upgrade does not change Parquet behavior. We can upgrade Parquet separately after testing it across Beam.
The existing Iceberg read/write APIs used by Beam are still available in Iceberg 1.11.0. They are marked as deprecated, but have not been removed, so no source-code changes are needed in this PR. A follow-up PR will migrate Beam to Iceberg’s newer API.
What was tested
Not run locally
Cloud-backed Iceberg integration tests require Google Cloud credentials.
Full linkage-checker output could not be completed locally because of build-infrastructure dependency resolution issues.
Reviewer decision requested (Dependency Questions)
ORC artifact selection: Iceberg uses the
orc-core:nohiveartifact. Beam currently declares unclassifiedorc-core, which can conflict with Iceberg’s ORC classes. The focused Beam ORC read test passes withnohiveon both ORC 1.9.6 and 1.9.8. Should this PR include the small dependency correction, or should it remain strictly scoped to the Iceberg bump and track that correction separately?Avro version: Beam deliberately forces Avro 1.12.0 even though Iceberg 1.11.0 requests 1.12.1. The current test suite passes with Beam’s resolved 1.12.0, so this PR does not change Avro behavior. However, the issue that motivated avoiding 1.12.1 looks to have been resolved AVRO-4209. Should we keep the existing pin in this PR and open a separate Beam-wide Avro 1.12.2 upgrade/validation PR?