Skip to content

Commit a1b64d5

Browse files
sbernauersbernauer
authored
feat(hadoop): Add 3.5.0 and 3.4.3 (#1511)
* feat: Add Hadoop 3.5.0 * revert accidential change * Bump HDFS dependants 3.4.2 -> 3.4.3 * Fix leftover EOF * Fix hadolint * WIP: Move s3a and abfs jar collection into hadoop builder image * Fixup aws-sdk file name * Add note * Switch to Nexus (as it now works again) * Remove parallel maven builds again * Fix another bundle name * (Hopefully) fix HBase build. Azure libs are already there * Downgrade Hadoop in Spark HBase-connector again * Downgrade HDFS in HBase as well * Use released hdfs-utils * Dont bump dependants * changelog
1 parent aabb6d0 commit a1b64d5

39 files changed

Lines changed: 1583 additions & 148 deletions

‎.scripts/upload_new_boost_version.sh‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/usr/bin/env bash
2+
# This script mirrors the boostorg/boost source bundle for the given version to Nexus.
3+
# The boost source bundle is architecture independent.
4+
# It contains its own build system (b2) which is also built from source before building boost itself, so we don't need to worry about architecture specific builds.
5+
# This artifact is used by the hadoop/boost local image.
6+
7+
8+
set -euo pipefail
9+
10+
VERSION=${1:?"Missing version number argument (arg 1)"}
11+
NEXUS_USER=${2:?"Missing Nexus username argument (arg 2)"}
12+
13+
read -r -s -p "Nexus Password: " NEXUS_PASSWORD
14+
echo ""
15+
16+
# https://stackoverflow.com/questions/4632028/how-to-create-a-temporary-directory
17+
# Find the directory name of the script
18+
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
19+
20+
# the temp directory used, within $DIR
21+
WORK_DIR=$(mktemp -d -p "$DIR")
22+
23+
# check if tmp dir was created
24+
if [[ ! "$WORK_DIR" || ! -d "$WORK_DIR" ]]; then
25+
echo "Could not create temp dir"
26+
exit 1
27+
fi
28+
29+
# deletes the temp directory
30+
function cleanup {
31+
rm -rf "$WORK_DIR"
32+
}
33+
34+
# register the cleanup function to be called on the EXIT signal
35+
trap cleanup EXIT
36+
37+
cd "$WORK_DIR" || exit
38+
39+
# boost does not currently publish signatures or SBOMs
40+
BOOST_UNDERSCORE="$(echo "${VERSION}" | tr '.' '_')"
41+
BOOST_TARBALL="boost_${BOOST_UNDERSCORE}.tar.bz2"
42+
DOWNLOAD_URL="https://archives.boost.io/release/$VERSION/source/$BOOST_TARBALL"
43+
44+
echo "Downloading boost"
45+
if ! curl --fail -Ls -O "$DOWNLOAD_URL"; then
46+
echo "Failed to download from $DOWNLOAD_URL"
47+
exit 1
48+
fi
49+
50+
FILE_NAME=$(basename "$DOWNLOAD_URL")
51+
52+
echo "Uploading boost to Nexus"
53+
if ! curl --fail -o /dev/null --progress-bar -u "$NEXUS_USER:$NEXUS_PASSWORD" --upload-file "$FILE_NAME" 'https://repo.stackable.tech/repository/packages/boost/'; then
54+
echo "Failed to upload boost to Nexus"
55+
exit 1
56+
fi
57+
58+
echo "Successfully uploaded new version of boost ($VERSION) to Nexus"
59+
echo "https://repo.stackable.tech/service/rest/repository/browse/packages/boost/"

‎CHANGELOG.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,9 @@ All notable changes to this project will be documented in this file.
1414
- opa: Add `1.16.2` ([#1509]).
1515
- kafka: Add `3.9.2` and `4.2.1` ([#1483]).
1616
- trino: Add `481` ([#1518]).
17-
- superset: Add `6.1.0` ([#1514])
18-
- zookeeper: Add `3.9.5` ([#1515])
17+
- superset: Add `6.1.0` ([#1514]).
18+
- zookeeper: Add `3.9.5` ([#1515]).
19+
- hadoop: Add `3.5.0` and `3.4.3` ([#1511]).
1920

2021
### Changed
2122

@@ -26,6 +27,7 @@ All notable changes to this project will be documented in this file.
2627
- statsd_exporter: Bump version from `0.28.0` to `0.30.0` ([#1524]).
2728
This uses a git mirror and patchable instead of sourcing from Nexus.
2829
- superset: Bump statsd_exporter to `0.30.0` ([#1524]).
30+
- hadoop: Refactor cloud library provisioning to downstream images. The jars are now conveniently placed in `/stackable/hadoop-cloud-libraries/` to be easily picked up ([#1511]).
2931

3032
### Fixed
3133

@@ -57,6 +59,7 @@ All notable changes to this project will be documented in this file.
5759
[#1507]: https://github.com/stackabletech/docker-images/pull/1507
5860
[#1509]: https://github.com/stackabletech/docker-images/pull/1509
5961
[#1510]: https://github.com/stackabletech/docker-images/pull/1510
62+
[#1511]: https://github.com/stackabletech/docker-images/pull/1511
6063
[#1512]: https://github.com/stackabletech/docker-images/pull/1512
6164
[#1514]: https://github.com/stackabletech/docker-images/pull/1514
6265
[#1515]: https://github.com/stackabletech/docker-images/pull/1515

‎druid/boil-config.toml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ authorizer-version = "0.7.0"
1515
# https://druid.apache.org/docs/34.0.0/operations/java/
1616
java-base = "17"
1717
java-devel = "17"
18-
"hadoop/hadoop" = "3.4.2"
18+
"hadoop/hadoop" = "3.4.2" # TODO: Bump to 3.4.3
1919

2020
# Deprecated since 26.3
2121
[versions."34.0.0".build-arguments]
@@ -26,7 +26,7 @@ authorizer-version = "0.7.0"
2626
# https://druid.apache.org/docs/35.0.1/operations/java/
2727
java-base = "21"
2828
java-devel = "21"
29-
"hadoop/hadoop" = "3.4.2"
29+
"hadoop/hadoop" = "3.4.2" # TODO: Bump to 3.4.3
3030

3131
[versions."35.0.1".build-arguments]
3232
authorizer-version = "0.7.0"

‎hadoop/boil-config.toml‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,24 @@ java-devel = "11"
2020
[versions."3.4.2".build-arguments]
2121
async-profiler-version = "2.9"
2222
jmx-exporter-version = "1.3.0"
23-
hdfs-utils-version = "0.5.0"
23+
hdfs-utils-version = "0.6.0"
24+
25+
[versions."3.4.3".local-images]
26+
"hadoop/hadoop" = "3.4.3"
27+
java-base = "11"
28+
java-devel = "11"
29+
30+
[versions."3.4.3".build-arguments]
31+
async-profiler-version = "2.9"
32+
jmx-exporter-version = "1.3.0"
33+
hdfs-utils-version = "0.6.0"
34+
35+
[versions."3.5.0".local-images]
36+
"hadoop/hadoop" = "3.5.0"
37+
java-base = "17"
38+
java-devel = "17"
39+
40+
[versions."3.5.0".build-arguments]
41+
async-profiler-version = "2.9"
42+
jmx-exporter-version = "1.3.0"
43+
hdfs-utils-version = "0.6.0"

‎hadoop/hadoop/Dockerfile‎

Lines changed: 53 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,15 @@
11
# syntax=docker/dockerfile:1.16.0@sha256:e2dd261f92e4b763d789984f6eab84be66ab4f5f08052316d8eb8f173593acf7
22
# check=error=true
33

4+
FROM local-image/shared/boost AS boost-builder
45
FROM local-image/java-devel AS hadoop-builder
56

67
ARG PRODUCT_VERSION
78
ARG RELEASE_VERSION
89
ARG PROTOBUF_VERSION
10+
ARG AWS_JAVA_SDK_BUNDLE_VERSION
11+
ARG AZURE_STORAGE_VERSION
12+
ARG AZURE_KEYVAULT_CORE_VERSION
913
ARG STACKABLE_USER_UID
1014

1115
WORKDIR /stackable
@@ -14,17 +18,22 @@ COPY --chown=${STACKABLE_USER_UID}:0 shared/protobuf/stackable/patches/patchable
1418
COPY --chown=${STACKABLE_USER_UID}:0 shared/protobuf/stackable/patches/${PROTOBUF_VERSION} /stackable/src/shared/protobuf/stackable/patches/${PROTOBUF_VERSION}
1519

1620
RUN <<EOF
17-
rpm --install --replacepkgs https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
1821
microdnf update
19-
# boost is a build dependency starting in Hadoop 3.4.0 if compiling native code
20-
# automake and libtool are required to build protobuf
21-
microdnf install boost1.78-devel automake libtool
22+
# libstdc++ is a runtime dependency for boost,
23+
# automake and libtool are needed to build protobuf
24+
microdnf install libstdc++ automake libtool
2225
microdnf clean all
2326
rm -rf /var/cache/yum
2427
mkdir /opt/protobuf
2528
chown ${STACKABLE_USER_UID}:0 /opt/protobuf
2629
EOF
2730

31+
COPY --chown=${STACKABLE_USER_UID}:0 --from=boost-builder /stackable/boost /stackable/boost
32+
33+
ENV BOOST_ROOT=/stackable/boost
34+
ENV LD_LIBRARY_PATH=/stackable/boost/lib
35+
ENV CPATH=/stackable/boost/include
36+
2837
USER ${STACKABLE_USER_UID}
2938
# This Protobuf version is the exact version as used in the Hadoop Dockerfile
3039
# See https://github.com/apache/hadoop/blob/trunk/dev-support/docker/pkg-resolver/install-protobuf.sh
@@ -35,10 +44,18 @@ RUN <<EOF
3544
# Create snapshot of the source code including custom patches
3645
tar -czf /stackable/protobuf-${PROTOBUF_VERSION}-src.tar.gz .
3746

38-
./autogen.sh
39-
./configure --prefix=/opt/protobuf
40-
make "-j$(nproc)"
41-
make install
47+
if [ "$PROTOBUF_VERSION" == "3.7.1" ] || [ "$PROTOBUF_VERSION" == "3.21.12" ]; then
48+
./autogen.sh
49+
./configure --prefix=/opt/protobuf
50+
make "-j$(nproc)"
51+
make install
52+
else
53+
# protobuf > 3.21 bundles abseil-cpp and utf8_range as git submodules
54+
git submodule update --init --recursive
55+
cmake -S . -B build -DCMAKE_POSITION_INDEPENDENT_CODE=ON -Dprotobuf_BUILD_TESTS=OFF
56+
cmake --build build --parallel "$(nproc)"
57+
cmake --install build --prefix /opt/protobuf
58+
fi
4259
(cd .. && rm -r ${PROTOBUF_VERSION})
4360
EOF
4461

@@ -126,3 +143,31 @@ rm -rf /stackable/.m2
126143
# Set correct groups; make sure only required artifacts for the final image are located in /stackable
127144
chmod -R g=u /stackable
128145
EOF
146+
147+
RUN <<EOF
148+
NEW_VERSION=${PRODUCT_VERSION}-stackable${RELEASE_VERSION}
149+
150+
# Place cloud libraries (for s3a:// and abfs://) in the /stackable/hadoop-cloud-libraries/ folder to
151+
# have one central place for dependant images to pull them from
152+
mkdir -p /stackable/hadoop-cloud-libraries/
153+
154+
if [[ "$PRODUCT_VERSION" == "3.3.6" || "$PRODUCT_VERSION" == "3.4.2" ]]; then
155+
cp /stackable/hadoop/share/hadoop/tools/lib/hadoop-aws-${NEW_VERSION}.jar /stackable/hadoop-cloud-libraries/
156+
# For some reason it was renamed from "aws-java-sdk-bundle" to "bundle"
157+
if [[ "$PRODUCT_VERSION" == "3.3.6" ]]; then
158+
cp /stackable/hadoop/share/hadoop/tools/lib/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_VERSION}.jar /stackable/hadoop-cloud-libraries/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_VERSION}.jar
159+
else
160+
cp /stackable/hadoop/share/hadoop/tools/lib/bundle-${AWS_JAVA_SDK_BUNDLE_VERSION}.jar /stackable/hadoop-cloud-libraries/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_VERSION}.jar
161+
fi
162+
cp /stackable/hadoop/share/hadoop/tools/lib/hadoop-azure-${NEW_VERSION}.jar /stackable/hadoop-cloud-libraries/
163+
cp /stackable/hadoop/share/hadoop/tools/lib/azure-storage-${AZURE_STORAGE_VERSION}.jar /stackable/hadoop-cloud-libraries/
164+
cp /stackable/hadoop/share/hadoop/tools/lib/azure-keyvault-core-${AZURE_KEYVAULT_CORE_VERSION}.jar /stackable/hadoop-cloud-libraries/
165+
else
166+
cp /stackable/hadoop/share/hadoop/common/lib/hadoop-aws-${NEW_VERSION}.jar /stackable/hadoop-cloud-libraries/
167+
# Starting with Hadoop 3.4.3 the aws-java-sdk-bundle is not included any more.
168+
curl -o /stackable/hadoop-cloud-libraries/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_VERSION}.jar --fail "https://repo.stackable.tech/repository/packages/aws/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE_VERSION}.jar"
169+
cp /stackable/hadoop/share/hadoop/common/lib/hadoop-azure-${NEW_VERSION}.jar /stackable/hadoop-cloud-libraries/
170+
cp /stackable/hadoop/share/hadoop/common/lib/azure-storage-${AZURE_STORAGE_VERSION}.jar /stackable/hadoop-cloud-libraries/
171+
cp /stackable/hadoop/share/hadoop/common/lib/azure-keyvault-core-${AZURE_KEYVAULT_CORE_VERSION}.jar /stackable/hadoop-cloud-libraries/
172+
fi
173+
EOF

‎hadoop/hadoop/boil-config.toml‎

Lines changed: 36 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,44 @@
11
[versions."3.3.6".local-images]
22
java-devel = "11"
3+
"shared/boost" = "1.72.0" # I could not find a documented recommended version
34

45
[versions."3.3.6".build-arguments]
5-
protobuf-version = "3.7.1"
6+
protobuf-version = "3.7.1" # https://github.com/apache/hadoop/blob/rel/release-3.3.6/BUILDING.txt
7+
aws-java-sdk-bundle-version = "1.12.367" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.3.6
8+
azure-storage-version = "7.0.1" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.3.6
9+
azure-keyvault-core-version = "1.0.0" # Needs to match https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1
10+
611

712
[versions."3.4.2".local-images]
8-
java-devel = "11"
13+
java-devel = "11" # https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
14+
"shared/boost" = "1.72.0" # https://github.com/apache/hadoop/blob/rel/release-3.4.2/BUILDING.txt
915

1016
[versions."3.4.2".build-arguments]
11-
protobuf-version = "3.7.1"
17+
protobuf-version = "3.21.12" # https://github.com/apache/hadoop/blob/rel/release-3.4.2/BUILDING.txt
18+
aws-java-sdk-bundle-version = "2.29.52" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.4.2
19+
azure-storage-version = "7.0.1" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.4.2
20+
azure-keyvault-core-version = "1.0.0" # Needs to match https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1
21+
22+
23+
24+
[versions."3.4.3".local-images]
25+
java-devel = "11" # https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
26+
"shared/boost" = "1.72.0" # https://github.com/apache/hadoop/blob/rel/release-3.4.3/BUILDING.txt
27+
28+
[versions."3.4.3".build-arguments]
29+
protobuf-version = "3.21.12" # https://github.com/apache/hadoop/blob/rel/release-3.4.3/BUILDING.txt
30+
aws-java-sdk-bundle-version = "2.35.4" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.4.3
31+
azure-storage-version = "7.0.1" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.4.3
32+
azure-keyvault-core-version = "1.0.0" # Needs to match https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1
33+
34+
35+
36+
[versions."3.5.0".local-images]
37+
java-devel = "17" # https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
38+
"shared/boost" = "1.86.0" # https://github.com/apache/hadoop/blob/rel/release-3.5.0/BUILDING.txt
39+
40+
[versions."3.5.0".build-arguments]
41+
protobuf-version = "3.25.5" # https://github.com/apache/hadoop/blob/rel/release-3.5.0/BUILDING.txt
42+
aws-java-sdk-bundle-version = "2.35.4" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.5.0
43+
azure-storage-version = "7.0.1" # Needs to match https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.5.0
44+
azure-keyvault-core-version = "1.0.0" # Needs to match https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1

‎hadoop/hadoop/stackable/patches/3.4.3/0001-YARN-11527-Update-node.js.patch‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
From 7ad31a922a9fbcecd884b4bdf5c416f6b0ea539e Mon Sep 17 00:00:00 2001
2+
From: Sebastian Bernauer <sebastian.bernauer@stackable.tech>
3+
Date: Tue, 26 May 2026 15:40:40 +0200
4+
Subject: YARN-11527-Update-node.js
5+
6+
---
7+
hadoop-project/pom.xml | 2 +-
8+
1 file changed, 1 insertion(+), 1 deletion(-)
9+
10+
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
11+
index 0813904f98a..f837b1f5201 100644
12+
--- a/hadoop-project/pom.xml
13+
+++ b/hadoop-project/pom.xml
14+
@@ -236,7 +236,7 @@
15+
<woodstox.version>5.4.0</woodstox.version>
16+
<nimbus-jose-jwt.version>10.4</nimbus-jose-jwt.version>
17+
<jcip-annotations.version>1.0-1</jcip-annotations.version>
18+
- <nodejs.version>v12.22.1</nodejs.version>
19+
+ <nodejs.version>v14.17.0</nodejs.version>
20+
<yarnpkg.version>v1.22.5</yarnpkg.version>
21+
<apache-ant.version>1.10.13</apache-ant.version>
22+
<jmh.version>1.20</jmh.version>

0 commit comments

Comments
 (0)