Skip to content

[improve][build] Upgrade async-http-client to 3.0.10#25023

Merged
lhotari merged 14 commits into
apache:masterfrom
merlimat:async-http-3
Jun 11, 2026
Merged

[improve][build] Upgrade async-http-client to 3.0.10#25023
lhotari merged 14 commits into
apache:masterfrom
merlimat:async-http-3

Conversation

@merlimat

@merlimat merlimat commented Nov 26, 2025 •
edited by lhotari
Loading

Copy link
Copy Markdown
Contributor

Motivation

Upgrade async-http-client from the legacy 2.x line to the current major release line 3.x (3.0.10, the latest release). The 2.x line is in maintenance mode, while 3.x is the actively developed line, so this keeps Pulsar on a supported version for picking up future fixes.

Note: CVE-2024-53990 has already been addressed on master (#23725 disabled the AsyncHttpClient CookieStore, #23732 upgraded to a patched 2.x release), so it is not the motivation for this change.

Modifications

Upgrade org.asynchttpclient:async-http-client from 2.15.0 to 3.0.10 and adapt to the API and dependency changes:

  • Timeout configuration methods (setConnectTimeout, setReadTimeout, setRequestTimeout, setPooledConnectionIdleTimeout) now take java.time.Duration instead of int millis — updated AsyncHttpConnector, HttpClient, ControlledClusterFailover, FlowBase and AuthenticationProviderOpenID accordingly.
  • TLS hostname verification is now applied explicitly: PulsarHttpAsyncSslEngineFactory takes an enableHostnameVerification flag and sets the HTTPS endpoint identification algorithm on the SSLEngine, since with AHC 3.x hostname verification is no longer applied to engines created by a custom SslEngineFactory.
  • AHC 3.x depends on com.sun.activation:jakarta.activation, which was not republished past 2.0.x and conflicts with jakarta.activation:jakarta.activation-api 2.1.x (see Replace unmaintained com.sun.activation:jakarta.activation with jakarta.activation-api + Angus Activation (or eliminate the dependency) AsyncHttpClient/async-http-client#2190). Instead of bundling it, a component metadata rule in pulsar.java-conventions replaces any dependency on it with jakarta.activation:jakarta.activation-api + org.eclipse.angus:angus-activation (the EE10 implementation). The rule applies to all modules, so no future dependency can reintroduce the conflicting artifact.
  • Pinned org.reactivestreams:reactive-streams to 1.0.4 in the version catalog, since AHC 3.x declares 1.0.3 and would otherwise downgrade it in the pulsar-shell distribution.
  • The async-http-client-netty-utils artifact no longer exists in 3.x.
  • Excluded AHC's io.netty:netty-codec dependency with a targeted component metadata rule: in Netty 4.2 it is an empty backwards-compatibility aggregator whose only net effect is adding the unused netty-codec-marshalling and netty-codec-protobuf modules; the codec modules AHC actually needs (netty-codec-base, netty-codec-compression) already come in through its netty-codec-http dependency. netty-codec-http2 is a separate, genuinely used AHC dependency (HTTP/2 support) and is kept.
  • LICENSE updates for both binary distributions: async-http-client 3.0.10, removal of the com.sun.activation:jakarta.activation jar, the netty-codec-http2 jar now bundled in the shell distribution, and org.jetbrains:annotations 26.1.0 (a new AHC 3.x transitive dependency) listed under its own "JetBrains Annotations" heading.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests: the admin client, client HTTP lookup, OAuth2 and OIDC authentication tests exercise the async-http-client integration. checkBinaryLicense verifies that the LICENSE files match the jars bundled in the server and shell distributions.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Dependency changes: async-http-client 2.15.0 → 3.0.10; com.sun.activation:jakarta.activation replaced by jakarta.activation:jakarta.activation-api 2.1.3 + org.eclipse.angus:angus-activation 2.0.2; org.reactivestreams:reactive-streams pinned to 1.0.4.

Documentation

  • doc <!-- Your PR contains doc changes. -->
  • doc-required <!-- Your PR changes impact docs and you will update later -->
  • doc-not-needed <!-- Your PR changes do not impact docs -->
  • doc-complete <!-- Docs have been already added -->

Matching PR in forked repository

PR in forked repository: <!-- ENTER URL HERE -->

@merlimat merlimat added this to the 4.2.0 milestone Nov 26, 2025
@merlimat merlimat requested a review from lhotari November 26, 2025 20:03
@merlimat merlimat self-assigned this Nov 26, 2025
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Nov 26, 2025
@lhotari

lhotari commented Nov 27, 2025

Copy link
Copy Markdown
Member

Regarding CVE-2024-53990, it's also fixed in 2.12.4.

@lhotari lhotari mentioned this pull request Mar 23, 2026
Closed
@nodece

nodece commented Apr 7, 2026

Copy link
Copy Markdown
Member

Please use version 3.0.8.

merlimat added a commit to merlimat/pulsar that referenced this pull request Apr 14, 2026
@merlimat
[improve][build] Upgrade Netty from 4.1.132.Final to 4.2.12.Final
d92305b 
This upgrade brings Pulsar onto the Netty 4.2 line in preparation for
the async-http-client 3.x upgrade (apache#25023), which transitively depends
on Netty 4.2 and cannot be landed while Pulsar force-pins Netty 4.1.

Netty 4.1 and 4.2 cannot co-exist on the classpath (same io.netty.*
package namespace), so the upgrade has to be done in a single step.
The Netty team asserts source/binary forward compatibility from 4.1
to 4.2 for regular API users:

  https://netty.io/news/2025/04/03/4-2-0.html
  https://github.com/netty/netty/wiki/Netty-4.2-Migration-Guide

Changes in this PR:

* gradle/libs.versions.toml:
  - Bump netty from 4.1.132.Final to 4.2.12.Final.
  - Drop the separate netty-iouring version (0.0.26.Final).
    io_uring has graduated from incubator
    (io.netty.incubator:netty-incubator-transport-*-io_uring) to a
    first-class Netty artifact
    (io.netty:netty-transport-{classes,native}-io_uring), now pinned
    to the same Netty version.

* pulsar-common/build.gradle.kts: Point the io_uring consumer at the
  renamed aliases.

* pulsar-common/.../EventLoopUtil.java: Netty 4.2 removed the
  dedicated IOUringEventLoopGroup class. io_uring now uses the generic
  MultiThreadIoEventLoopGroup + IoUringIoHandler factory pattern,
  which makes io_uring groups indistinguishable from any other
  MultiThreadIoEventLoopGroup by type, breaking the existing
  instanceof-based channel class dispatch. Fix: introduce a private
  marker subclass IoUringMultiThreadIoEventLoopGroup used at
  construction. Also repoint the incubator imports
  (io.netty.incubator.channel.uring.*) to the core package
  (io.netty.channel.uring.*) and adjust class names (IOUring ->
  IoUring).

* build-logic/conventions/.../pulsar.java-conventions.gradle.kts:
  Exclude io.netty.incubator from all configurations. BookKeeper 4.17.3
  (bookkeeper-common and stream-storage-java-client) still declares a
  transitive dependency on the 0.0.26.Final incubator io_uring jars,
  which are compiled against Netty 4.1 internals and are not safe to
  leave on the 4.2 classpath. Pulsar uses the core io_uring API via
  EventLoopUtil; BK stream-storage is an optional feature that Pulsar
  does not expose in its default surface.

* distribution/{server,shell}/src/assemble/LICENSE.bin.txt: Reflect
  the actual Netty jar set shipped after the upgrade:
  - Bump all 4.1.132.Final entries to 4.2.12.Final.
  - Replace the monolithic netty-codec-*.jar with its 4.2 split-out
    sub-modules netty-codec-base and netty-codec-compression
    (netty-codec is now an aggregator POM that ships no classes).
  - Rename the incubator io_uring entries
    (io.netty.incubator-netty-incubator-transport-*-io_uring-0.0.26.Final)
    to the core io_uring artifacts
    (io.netty-netty-transport-{classes,native}-io_uring-4.2.12.Final).
  The jar set was cross-checked against the output of
  :distribution:pulsar-server-distribution:serverDistTar and
  :distribution:pulsar-shell-distribution:shellDistTar.

* pulsar-common/.../BitSetRecyclableRecyclableTest and
  ConcurrentBitSetRecyclableTest: Relax the testRecycle assertion.
  Netty 4.2's io.netty.util.Recycler (which is itself deprecated in
  4.2) no longer guarantees same-thread immediate reuse, so we only
  assert functional behavior: any recycled instance must come back
  cleared, and distinct create() calls must return distinct objects.

Verification:

* ./gradlew compileJava compileTestJava: clean across the entire
  project, only deprecation warnings (NioEventLoopGroup,
  EpollEventLoopGroup, DefaultEventLoopGroup,
  ChannelOption.RCVBUF_ALLOCATOR, EpollMode, Recycler,
  PlatformDependent.threadLocalRandom). These are compat shims that
  still function in 4.2; cleanup can follow in a separate PR.

* :pulsar-common:test: passes (678 tests).
* :pulsar-broker:test --tests BrokerServiceTest: passes (broker
  startup, producer/consumer flow, Netty transport end-to-end).
* :pulsar-proxy:test --tests ProxyServiceTlsStarterTest: passes
  (proxy, TLS handshake, tcnative-boringssl integration).
* :distribution:pulsar-server-distribution:serverDistTar and
  :distribution:pulsar-shell-distribution:shellDistTar both build,
  and the Netty jar set inside each tarball matches the
  LICENSE.bin.txt files exactly.

Known Netty 4.2 behavior changes that this PR does NOT address:

* The default SslContextBuilder.endpointIdentificationAlgorithm
  changed from null to HTTPS in 4.2. Pulsar's TLS client sites need
  to be audited and explicitly configured. This is intentionally out
  of scope here because the audit touches many modules
  (pulsar-client, pulsar-broker, pulsar-proxy,
  pulsar-broker-auth-oidc, admin) and should be its own PR.

* The default ByteBufAllocator changed from pooled to adaptive in
  4.2. Pulsar is not setting io.netty.allocator.type=pooled in this
  PR; if CI soak tests show regressions, the pooled override can be
  added to the launch scripts as a follow-up.
@merlimat merlimat mentioned this pull request Apr 14, 2026
Closed
@merlimat

merlimat commented Apr 23, 2026

Copy link
Copy Markdown
Contributor Author

Deferring this upgrade until BK 4.18, which will also be using Netty 4.2.x

@dao-jun

dao-jun commented Apr 24, 2026

Copy link
Copy Markdown
Member

Deferring this upgrade until BK 4.18, which will also be using Netty 4.2.x

@merlimat
apache/bookkeeper#4739
apache/bookkeeper#4741

I want the 2 PRs can be released in BK 4.18, could you please take a look?

@merlimat

merlimat commented Apr 24, 2026

Copy link
Copy Markdown
Contributor Author

I want the 2 PRs can be released in BK 4.18, could you please take a look?

Yes, no worries, we're still a few weeks away from BK 4.18. A discussion will be started before starting.

@lhotari lhotari modified the milestones: 4.2.0, 4.3.0 Apr 27, 2026
@lhotari
Merge branch 'master' into async-http-3
f77c144 
Conflict resolutions and adaptations for the Gradle build migration on
master:

- Removed pom.xml and pulsar-broker/pom.xml (build migrated to Gradle on
  master); ported the async-http-client upgrade to gradle/libs.versions.toml
  and bumped it to 3.0.10.
- Replaced com.sun.activation:jakarta.activation (pulled in by
  async-http-client, see
  AsyncHttpClient/async-http-client#2190) with
  jakarta.activation:jakarta.activation-api + org.eclipse.angus:angus-activation
  via a component metadata rule in pulsar.java-conventions so that no future
  dependency can reintroduce it.
- Pinned org.reactivestreams:reactive-streams to 1.0.4 so async-http-client's
  1.0.3 dependency does not downgrade it in the shell distribution.
- AsyncHttpConnector: kept the AHC 3.x PulsarHttpAsyncSslEngineFactory based
  hostname verification from this branch and master's SOCKS5 proxy support.
- FlowBase: kept master's refactored duration helpers/structured logging,
  updated the PulsarHttpAsyncSslEngineFactory call to the 3-arg constructor
  (hostname verification enabled, matching previous AHC defaults) and removed
  the now-unused getParameterDurationToMillis.
- LICENSE.bin.txt: async-http-client 3.0.10 (netty-utils artifact no longer
  exists), no more com.sun.activation jar, added netty-codec/marshalling/
  protobuf/http2 jars now bundled via AHC's netty-codec dependency, and
  listed org.jetbrains annotations 26.1.0 under its own "JetBrains
  Annotations" heading instead of "Kotlin Standard Lib".
@lhotari lhotari changed the title [fix] Upgrade to AsyncHTTP client 3.0.4 [improve][build] Upgrade async-http-client to 3.0.10 Jun 11, 2026
@lhotari
Exclude unused netty-codec aggregator pulled in by async-http-client
f185d39 
In Netty 4.2, io.netty:netty-codec is an empty backwards-compatibility
aggregator whose only effect on the classpath is adding the unused
netty-codec-marshalling and netty-codec-protobuf modules. The codec
modules async-http-client actually needs (netty-codec-base,
netty-codec-compression) already come in through its netty-codec-http
dependency, so remove the netty-codec dependency edge with a component
metadata rule and drop the three jars from the distribution LICENSE
files. netty-codec-http2 is a separate, genuinely used dependency of
async-http-client and is kept.
@lhotari lhotari merged commit 140e139 into apache:master Jun 11, 2026
81 of 84 checks passed
@merlimat merlimat deleted the async-http-3 branch June 11, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lhotari lhotari lhotari approved these changes
@nodece nodece nodece approved these changes
@dao-jun dao-jun dao-jun approved these changes

Assignees

@merlimat merlimat

Labels

doc-not-needed Your PR changes do not impact docs ready-to-test

Projects

None yet

Milestone

5.0.0-M1

Development

Successfully merging this pull request may close these issues.

4 participants

@merlimat @lhotari @nodece @dao-jun