Skip to content

chore: Updating spark-k8s dependency to hadoop 3.4.3 #1533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Maleware merged 9 commits into from
Jun 16, 2026
Merged

chore: Updating spark-k8s dependency to hadoop 3.4.3 #1533

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
5c52e48
Updating spark-k8s dependency to hadoop 3.4.3
Maleware Jun 15, 2026
d9930cc
Merge branch 'main' into chore/spark-update-hadoop-version
Maleware Jun 15, 2026
5d4b247
Changing from mvn to sble repo
Maleware Jun 15, 2026
04e5ae8
Merge branch 'chore/spark-update-hadoop-version' of https://github.co…
Maleware Jun 15, 2026
aaf2a19
Update changelog.md
Maleware Jun 15, 2026
278801c
fixing jackson version mismatch
Maleware Jun 15, 2026
593a134
apply review formatting
Maleware Jun 16, 2026
8161362
Apply review, spacing and link
Maleware Jun 16, 2026
cd39aa4
Fixing comments part2
Maleware Jun 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ All notable changes to this project will be documented in this file.
This uses a git mirror and patchable instead of sourcing from Nexus.
- superset: Bump statsd_exporter to `0.30.0` ([#1524]).
- hadoop: Refactor cloud library provisioning to downstream images. The jars are now conveniently placed in `/stackable/hadoop-cloud-libraries/` to be easily picked up ([#1511]).
- spark: bump hadoop `3.4.2` to `3.4.3` ([#1533])

### Fixed

Expand Down Expand Up @@ -68,8 +69,9 @@ All notable changes to this project will be documented in this file.
[#1519]: https://github.com/stackabletech/docker-images/pull/1519
[#1520]: https://github.com/stackabletech/docker-images/pull/1520
[#1521]: https://github.com/stackabletech/docker-images/pull/1521
[#1525]: https://github.com/stackabletech/docker-images/pull/1525
[#1524]: https://github.com/stackabletech/docker-images/pull/1524
[#1525]: https://github.com/stackabletech/docker-images/pull/1525
[#1533]: https://github.com/stackabletech/docker-images/pull/1533

## [26.3.0] - 2026-03-16

Expand Down
3 changes: 3 additions & 0 deletions hadoop/hadoop/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ ARG PROTOBUF_VERSION
ARG AWS_JAVA_SDK_BUNDLE_VERSION
ARG AZURE_STORAGE_VERSION
ARG AZURE_KEYVAULT_CORE_VERSION
ARG ANALYTICSACCELERATOR_S3_VERSION
ARG STACKABLE_USER_UID

WORKDIR /stackable
Expand Down Expand Up @@ -166,6 +167,8 @@ else
cp /stackable/hadoop/share/hadoop/common/lib/hadoop-aws-${NEW_VERSION}.jar /stackable/hadoop-cloud-libraries/
# Starting with Hadoop 3.4.3 the aws-java-sdk-bundle is not included any more.
curl -o /stackable/hadoop-cloud-libraries/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_VERSION}.jar --fail "https://repo.stackable.tech/repository/packages/aws/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_VERSION}.jar"
# Starting with Hadoop 3.4.3 the S3A InputStreamType factory references analyticsaccelerator-s3 (software.amazon.s3.analyticsaccelerator.request.ObjectClient) during S3AFileSystem.initialize.
curl -o /stackable/hadoop-cloud-libraries/analyticsaccelerator-s3-${ANALYTICSACCELERATOR_S3_VERSION}.jar --fail "https://repo.stackable.tech/repository/packages/aws/analyticsaccelerator-s3-${ANALYTICSACCELERATOR_S3_VERSION}.jar"
cp /stackable/hadoop/share/hadoop/common/lib/hadoop-azure-${NEW_VERSION}.jar /stackable/hadoop-cloud-libraries/
cp /stackable/hadoop/share/hadoop/common/lib/azure-storage-${AZURE_STORAGE_VERSION}.jar /stackable/hadoop-cloud-libraries/
cp /stackable/hadoop/share/hadoop/common/lib/azure-keyvault-core-${AZURE_KEYVAULT_CORE_VERSION}.jar /stackable/hadoop-cloud-libraries/
Expand Down
4 changes: 2 additions & 2 deletions hadoop/hadoop/boil-config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,7 @@ protobuf-version = "3.21.12" # https://github.com/apache/hadoop/blob/rel/release
aws-java-sdk-bundle-version = "2.35.4" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.4.3
azure-storage-version = "7.0.1" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.4.3
azure-keyvault-core-version = "1.0.0" # Needs to match https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1


analyticsaccelerator-s3-version = "1.3.1" # Pinned in hadoop-project-3.4.3.pom via amazon-s3-analyticsaccelerator-s3.version.
Comment thread
Maleware marked this conversation as resolved.
Outdated

[versions."3.5.0".local-images]
java-devel = "17" # https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
Expand All @@ -42,3 +41,4 @@ protobuf-version = "3.25.5" # https://github.com/apache/hadoop/blob/rel/release-
aws-java-sdk-bundle-version = "2.35.4" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.5.0
azure-storage-version = "7.0.1" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.5.0
azure-keyvault-core-version = "1.0.0" # Needs to match https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1
analyticsaccelerator-s3-version = "1.3.1" # Pinned in hadoop-project-3.5.0.pom via amazon-s3-analyticsaccelerator-s3.version.
Comment thread
sbernauer marked this conversation as resolved.
Outdated
10 changes: 5 additions & 5 deletions spark-k8s/boil-config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
containerfile = "Dockerfile.3"

[versions."3.5.8".local-images]
"hadoop/hadoop" = "3.4.2" # TODO: Bump to 3.4.3
"hadoop/hadoop" = "3.4.3"
java-base = "17"
java-devel = "17"
hbase = "2.6.3"
Expand All @@ -27,14 +27,14 @@ hbase-connector-version = "1.0.1_3.5.8"
containerfile = "Dockerfile.4"

[versions."4.1.1".local-images]
"hadoop/hadoop" = "3.4.2" # TODO: Bump to 3.4.3
"hadoop/hadoop" = "3.4.3"
java-base = "21"
java-devel = "21"

[versions."4.1.1".build-arguments]
python-version = "3.12"
jackson-dataformat-xml-version = "2.15.2" # Needs to match https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.5.7
Comment thread
sbernauer marked this conversation as resolved.
stax2-api-version = "4.2.1" # Needs to match https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2
woodstox-core-version = "6.5.1" # Needs to match https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2
jackson-dataformat-xml-version = "2.20.0" # Needs to match fasterxml.jackson.version in https://mvnrepository.com/artifact/org.apache.spark/spark-parent_2.13/4.1.1
stax2-api-version = "4.2.2" # Needs to match jackson.version.dep.stax2-api in jackson-bom-2.20.0
woodstox-core-version = "7.1.1" # Needs to match jackson.version.dep.woodstox in jackson-bom-2.20.0
jmx-exporter-version = "1.3.0"
tini-version = "0.19.0"