Skip to content

feat!: Add support for 3.5.0, remove older versions #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sbernauer merged 9 commits into from
Jun 9, 2026
Merged

feat!: Add support for 3.5.0, remove older versions #62

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7b0817a
feat!: Add support for 3.5.0, remove older versions
sbernauer May 26, 2026
c44b056
Random other bumps
sbernauer May 26, 2026
d0c88b5
chore: Bump to 0.6.0
sbernauer May 27, 2026
e3708f1
Downgrade to Java 11
sbernauer May 27, 2026
b3ebd85
fix: Shade jackson libs
sbernauer May 27, 2026
3bcf5a0
Stupid markdown linter
sbernauer May 27, 2026
f037a42
Update CHANGELOG.md
sbernauer Jun 2, 2026
5f916dc
Update CHANGELOG.md
sbernauer Jun 2, 2026
ad3978a
Adjust release date
sbernauer Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,18 @@ All notable changes to this project will be documented in this file.

## [Unreleased]

## [0.6.0] - 2026-??-??
Comment thread
sbernauer marked this conversation as resolved.
Outdated

### Added

- Added support Hadoop `3.5.0` ([#62]).

### Removed

- Removed support for Hadoop `3.3.6` and `3.4.1` ([#62]).

[#62]: https://github.com/stackabletech/hdfs-utils/pull/62

## [0.5.0] - 2026-02-24

### Changed
Expand Down
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ This project contains multiple plugins for Apache Hadoop, which are intended to

## Installation

Currently you can compile hdfs-utils against Hadoop 3.3.6, 3.4.1 or 3.4.2. You need to specify the version by activating either the `hadoop-3.3.6`, `hadoop-3.4.1` or the `hadoop-3.4.2` profile below.
Currently you can compile hdfs-utils against Hadoop 3.4.2, 3.4.3 or 3.5.0. You need to specify the version by activating either the `hadoop-3.4.2`, `hadoop-3.4.3` or the `hadoop-3.5.0` profile below.

Run e.g. `mvn clean package -P hadoop-3.4.2` and put the resulting `target/hdfs-utils-*.jar` file on your HDFS classpath.
Run e.g. `mvn clean package -P hadoop-3.5.0` and put the resulting `target/hdfs-utils-*.jar` file on your HDFS classpath.
The easiest way to achieve this is to put it in the directory `/stackable/hadoop/share/hadoop/tools/lib/`.
The Stackable HDFS already takes care of this, you don't need to do anything in this case.

Expand All @@ -21,6 +21,7 @@ The Stackable HDFS already takes care of this, you don't need to do anything in
> [!IMPORTANT]
> The authorizer only works when used by an HDFS version that includes fixes from https://github.com/apache/hadoop/pull/6553.
> Stackable HDFS versions starting with `3.3.4` already contain this patch.
> Upstream Hadoop fixed this in `3.5.0` via [this PR](https://github.com/apache/hadoop/pull/6776).

### Configuration

Expand Down
121 changes: 71 additions & 50 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

<groupId>tech.stackable</groupId>
<artifactId>hdfs-utils</artifactId>
<version>0.5.0</version>
<version>0.6.0</version>

<name>Apache Hadoop HDFS utils</name>
<url>https://github.com/stackabletech/hdfs-utils/</url>
Expand All @@ -29,6 +29,8 @@
</issueManagement>

<properties>
<!-- Hadoop >= 3.5.x is compiled against Java 17, 3.4.x against Java 11 -->
<!-- Once we drop HDFS 3.4.x support, we can bump to 17 here -->
<java.version>11</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
Expand All @@ -43,23 +45,44 @@
<error-prone.version>2.35.1</error-prone.version>
<google-java-format.version>1.24.0</google-java-format.version>

<cyclonedx-maven-plugin.version>2.9.0</cyclonedx-maven-plugin.version>
<maven-clean-plugin.version>3.4.0</maven-clean-plugin.version>
<maven-compiler-plugin.version>3.13.0</maven-compiler-plugin.version>
<maven-deploy-plugin.version>3.1.3</maven-deploy-plugin.version>
<maven-enforcer-plugin.version>3.5.0</maven-enforcer-plugin.version>
<maven-install-plugin.version>3.1.3</maven-install-plugin.version>
<maven-jar-plugin.version>3.4.2</maven-jar-plugin.version>
<maven-resources-plugin.version>3.3.1</maven-resources-plugin.version>
<maven-shade-plugin.version>3.6.0</maven-shade-plugin.version>
<cyclonedx-maven-plugin.version>2.9.1</cyclonedx-maven-plugin.version>
<maven-clean-plugin.version>3.5.0</maven-clean-plugin.version>
<maven-compiler-plugin.version>3.15.0</maven-compiler-plugin.version>
<maven-deploy-plugin.version>3.1.4</maven-deploy-plugin.version>
<maven-enforcer-plugin.version>3.6.3</maven-enforcer-plugin.version>
<maven-install-plugin.version>3.1.4</maven-install-plugin.version>
<maven-jar-plugin.version>3.5.0</maven-jar-plugin.version>
<maven-resources-plugin.version>3.5.0</maven-resources-plugin.version>
<maven-shade-plugin.version>3.6.2</maven-shade-plugin.version>
<maven-site-plugin.version>3.12.1</maven-site-plugin.version>
<maven-surefire-plugin.version>3.5.2</maven-surefire-plugin.version>
<spotless-maven-plugin.version>2.43.0</spotless-maven-plugin.version>
<maven-surefire-plugin.version>3.5.5</maven-surefire-plugin.version>
<spotless-maven-plugin.version>3.5.1</spotless-maven-plugin.version>

<kubernetes-client.version>6.13.1</kubernetes-client.version>
<okio.version>1.17.6</okio.version>
<kubernetes-client.version>7.7.0</kubernetes-client.version>
<!--
Jackson is bundled and relocated (see maven-shade-plugin), so it is decoupled from the
Jackson that Hadoop ships. Keep this aligned with the version the kubernetes-client pulls in
(jackson-datatype-jsr310 / jackson-dataformat-yaml) so the bundled Jackson set is consistent. -->
<jackson.version>2.21.3</jackson.version>
</properties>

<dependencyManagement>
<dependencies>
<!--
Force every Jackson module (databind, core, annotations, jsr310, yaml) to a single
version. Otherwise Maven mediation pins some modules (e.g. jackson-core) to the old
version pulled in through Hadoop's (provided) dependency tree, leaving the bundled and
relocated Jackson set internally inconsistent. -->
<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
<version>${jackson.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
Expand All @@ -79,17 +102,21 @@
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<!--
Bundled (not provided) and relocated via the shade plugin, see the relocations there.
Pinned to the version used by the fabric8 kubernetes-client so the whole Jackson set
(databind, core, annotations, jsr310, yaml) stays internally consistent, independent of
whatever Jackson the target Hadoop ships. -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
<scope>provided</scope>
</dependency>
<!-- Needed by topology-provider -->
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
<version>3.1.8</version>
<version>3.2.4</version>
</dependency>
<dependency>
<groupId>io.fabric8</groupId>
Expand All @@ -101,17 +128,6 @@
<artifactId>kubernetes-client-api</artifactId>
<version>${kubernetes-client.version}</version>
</dependency>
<dependency>
<!--
We bump this here to get rid of a critical CVE in okio 1.15 which we get via kubernetes-client.
We tried understanding _why_ we get 1.15 as dependency:tree for kubernetes-client says we should be getting 1.17.6.
As we failed to understand this we did this short/medium term fix of adding an explicit dependency here which should override the one coming from kubernetes-client.
This can be removed again as soon as we get the proper version from kubernetes-client.
-->
<groupId>com.squareup.okio</groupId>
<artifactId>okio</artifactId>
<version>${okio.version}</version>
</dependency>
<!-- End of needed by topology-provider -->
<dependency>
<groupId>junit</groupId>
Expand Down Expand Up @@ -204,6 +220,20 @@
<goal>shade</goal>
</goals>
<phase>package</phase>
<configuration>
<relocations>
<!--
Relocate Jackson so the bundled (fabric8 kubernetes-client) Jackson is fully
self-contained and does not clash with the Jackson that Hadoop ships on its
classpath. Without this, the NameNode loads Hadoop's old jackson-databind
(e.g. 2.12.7 on HDFS 3.4.x) while our bundled jackson-datatype-jsr310 expects a
newer one, causing NoClassDefFoundError (e.g. ToEmptyObjectSerializer). -->
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>tech.stackable.shaded.com.fasterxml.jackson</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
Expand Down Expand Up @@ -285,42 +315,33 @@

<profiles>
<profile>
<id>hadoop-3.3.6</id>
<properties>
<hadoop.version>3.3.6</hadoop.version>
<!-- Taken from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.3.6 -->
<slf4j.version>1.7.36</slf4j.version>
<jackson.version>2.12.7.1</jackson.version>
</properties>
</profile>
<profile>
<id>hadoop-3.4.1</id>
<id>hadoop-3.4.2</id>
<!--
Users need to explicitly set the version of Hadoop they want to build against
<activation>
<activeByDefault>true</activeByDefault>
</activation>
-->
<properties>
<hadoop.version>3.4.1</hadoop.version>
<!-- Taken from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.4.1 -->
<hadoop.version>3.4.2</hadoop.version>
<!-- Taken from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.4.2 -->
<slf4j.version>1.7.36</slf4j.version>
<jackson.version>2.12.7.1</jackson.version>
</properties>
</profile>
<profile>
<id>hadoop-3.4.2</id>
<!--
Users need to explicitly set the version of Hadoop they want to build against
<activation>
<activeByDefault>true</activeByDefault>
</activation>
-->
<id>hadoop-3.4.3</id>
<properties>
<hadoop.version>3.4.2</hadoop.version>
<!-- Taken from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.4.2 -->
<hadoop.version>3.4.3</hadoop.version>
<!-- Taken from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.4.3 -->
<slf4j.version>1.7.36</slf4j.version>
</properties>
</profile>
<profile>
<id>hadoop-3.5.0</id>
<properties>
<hadoop.version>3.5.0</hadoop.version>
<!-- Taken from https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.5.0 -->
<slf4j.version>1.7.36</slf4j.version>
<jackson.version>2.12.7.1</jackson.version>
</properties>
</profile>
</profiles>
Expand Down
Loading