Skip to content

Commit b8d77b8

Browse files
peterphitranAbacn
andauthored
[IcebergIO] Upgrade Iceberg dependency to 1.11.0 (#39559)
* [IcebergIO] Upgrade Iceberg dependency to 1.11.0 Bumps iceberg-core/api/parquet/orc/data and the GCP/AWS/Azure runtime modules from 1.10.0 to 1.11.0. Iceberg 1.11.0 pulls parquet-avro/parquet-hadoop 1.17.1 transitively; pin them at 1.16.0 (current Beam parquet_version) so this PR stays zero-behavior-change. A separate PR can bump parquet_version once 1.17.x is vetted across the rest of Beam. The old per-format static builders (Parquet.read(), Avro.writeData(), ORC.write(), etc.) are only deprecated in 1.11.0, not removed, so RecordWriter / ScanTaskReader / ReadUtils still compile unchanged. A follow-up PR will migrate them to the new FormatModelRegistry SPI. Rebased on merged #39064 (Java 17 floor). Tracks #38925. * Update changelog * Apply suggestions from code review Co-authored-by: Yi Hu <huuyyi@gmail.com> --------- Co-authored-by: Yi Hu <huuyyi@gmail.com>
1 parent 03db1a0 commit b8d77b8

2 files changed

Lines changed: 7 additions & 2 deletions

File tree

CHANGES.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@
6464

6565
## I/Os
6666

67+
* Upgraded Iceberg dependency to 1.11.0 (Java) ([#38925](https://github.com/apache/beam/issues/38925)).
6768
* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
6869
* Add ArrowFlight IO (Java) ([#20116](https://github.com/apache/beam/issues/20116)).
6970

@@ -85,7 +86,7 @@
8586
* (Python) Typehints of dataclass fields are honored during type inferences. To restore the behavior of fallback-to-any,
8687
use pipeline option `--exclude_infer_dataclass_field_type` ([#38797](https://github.com/apache/beam/issues/38797)).
8788
However fixing forward is recommended.
88-
* (Java) IcebergIO now requires Java 17 at runtime. This raises the floor in preparation for the Iceberg 1.11.0 upgrade ([#38925](https://github.com/apache/beam/issues/38925)).
89+
* (Java) IcebergIO and projects that use it must now be built with Java 17 or later as a result of Iceberg 1.11.0 upgrade ([#38925](https://github.com/apache/beam/issues/38925)).
8990

9091
## Deprecations
9192

sdks/java/io/iceberg/build.gradle

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ def hadoopVersions = [
3737

3838
hadoopVersions.each {kv -> configurations.create("hadoopVersion$kv.key")}
3939

40-
def iceberg_version = "1.10.0"
40+
def iceberg_version = "1.11.0"
4141
def parquet_version = "1.16.0"
4242
def orc_version = "1.9.6"
4343
def hive_version = "3.1.3"
@@ -118,6 +118,10 @@ dependencies {
118118
configurations.all {
119119
// iceberg-core needs avro:1.12.0
120120
resolutionStrategy.force 'org.apache.avro:avro:1.12.0'
121+
// Iceberg 1.11.0 pulls parquet 1.17.1 transitively; hold at 1.16.0 to keep
122+
// this PR zero-behavior-change. Bump parquet_version in a separate PR.
123+
resolutionStrategy.force 'org.apache.parquet:parquet-avro:1.16.0'
124+
resolutionStrategy.force 'org.apache.parquet:parquet-hadoop:1.16.0'
121125
// TODO(https://github.com/apache/beam/issues/38515):
122126
// Remove below pins when parquet-hadoop upgrades to hadoop-common:3.4.2
123127
resolutionStrategy.force 'org.apache.hadoop:hadoop-common:3.3.6'

0 commit comments

Comments
 (0)