StreamResetException in Vert.x HTTP client when LastHttpContent arrives before HttpResponse in Netty pipeline (HttpClientRequestTracingHandler)

[jvs87]

Describe the bug

When using the OpenTelemetry Java agent (2.16.0-alpha) with a Vert.x HTTP client making outbound POST requests, the request fails with a StreamResetException: Stream reset: 0, causing the parent span to be marked as HTTP 500 ERROR.
The root cause is a java.lang.IllegalStateException: unexpected message type: LastHttpContent$1, state: 0 thrown by Netty's HttpObjectEncoder. This happens because LastHttpContent reaches the encoder when the connection is in state: 0 (idle/uninitialized), meaning no prior HttpRequest was processed on that channel — which suggests a pooled connection was reused after being closed by the server, and the HttpClientRequestTracingHandler forwarded the write into a pipeline that was no longer in a valid state.
The HttpClientRequestTracingHandler appears in the stacktrace at line 40, between CombinedChannelDuplexHandler and HttpObjectEncoder, indicating the OTel instrumentation handler is part of the failing write chain.

io.vertx.core.http.StreamResetException: Stream reset: 0 Caused by: io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: LastHttpContent$1, state: 0 at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:111) at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346) ... at io.opentelemetry.javaagent.shaded.instrumentation.netty.v4_1.internal.client.HttpClientRequestTracingHandler.write(HttpClientRequestTracingHandler.java:40) at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346) ... Caused by: java.lang.IllegalStateException: unexpected message type: LastHttpContent$1, state: 0 at io.netty.handler.codec.http.HttpObjectEncoder.throwUnexpectedMessageTypeEx(HttpObjectEncoder.java:364) at io.netty.handler.codec.http.HttpObjectEncoder.encodeNotHttpMessageContentTypes(HttpObjectEncoder.java:270) at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:184) at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:290) at io.netty.handler.codec.http.HttpObjectEncoder.write(HttpObjectEncoder.java:100)

Steps to reproduce

The issue occurs in a production environment and we have not been able to create a minimal reproducer yet, but the conditions observed in the distributed trace are:

Vert.x application starts with the OTel Java agent attached:

java -javaagent:opentelemetry-javaagent-2.16.0-alpha.jar
-jar app.jar

An inbound POST /bc-authorize request is received by the Vert.x server.
The service first makes a successful outbound GET request via OkHttp to orgs..com/v2/msisdns/{msisdn} (HTTP 200).
The service then attempts an outbound POST via the Vert.x HTTP client (Http1xClientConnection) to authn..com/v1/bc-authorize over HTTP/1.1, reusing a pooled connection.
The pooled connection is in an invalid state (closed by the server or expired), but the Vert.x connection pool returns it as valid.
The HttpClientRequestTracingHandler (OTel) forwards the write into the Netty pipeline. LastHttpContent reaches HttpObjectEncoder before any HttpRequest has been processed on that channel (state: 0).
Netty throws IllegalStateException: unexpected message type: LastHttpContent$1, state: 0, which Vert.x wraps as StreamResetException: Stream reset: 0.
The parent span POST /bc-authorize is marked as HTTP 500 ERROR.

Note: The issue appears to be intermittent and related to connection pool reuse under load in a Kubernetes environment. Removing the OTel agent from the JVM arguments is expected to change the behavior, as HttpClientRequestTracingHandler would no longer be present in the Netty pipeline write chain.

Expected behavior

The outbound POST request to authn.****.com/v1/bc-authorize should either:

Complete successfully if the connection is still valid, or
Fail gracefully with a connection error (e.g. ConnectException or a clean HTTP error response) if the pooled connection is stale — without throwing an IllegalStateException inside the Netty pipeline write chain.

In either case, the HttpClientRequestTracingHandler should not interfere with the Netty pipeline state or cause a StreamResetException that would not occur without the OTel agent attached. The instrumentation handler should be transparent to the underlying HTTP encoding state machine.
This behavior is consistent with what PR #16879 addresses in the testing module: when LastHttpContent arrives before an HttpResponse has been stored, the handler should fail explicitly and cleanly (completeExceptionally) rather than propagating an unhandled exception through the pipeline.

Actual behavior

The outbound POST request to authn.****.com/v1/bc-authorize fails with a StreamResetException: Stream reset: 0, which propagates up and causes the parent span POST /bc-authorize to return HTTP 500 with otel.status_code: ERROR.
The failure originates inside the Netty pipeline write chain, where HttpClientRequestTracingHandler (OTel, line 40) forwards a LastHttpContent frame to HttpObjectEncoder while the encoder is in state: 0 (no active request). This state indicates the connection was stale or already closed, but the Vert.x HTTP client pool returned it as valid and the OTel handler did not detect or handle this condition before forwarding the write.
The exception chain observed in the trace is:

io.vertx.core.http.StreamResetException: Stream reset: 0 Caused by: io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: LastHttpContent$1, state: 0

This error would not be expected to occur — or would manifest differently — without the OTel agent in the JVM, since HttpClientRequestTracingHandler would not be present in the pipeline write chain. The instrumentation handler introduces a layer that disrupts the normal Netty HTTP encoding state machine when the underlying connection is in an invalid state.

Javaagent or library instrumentation version

opentelemetry-javaagent.jar )2.23.0

Environment

• telemetry.sdk.version: 1.50.0
• telemetry.distro.name: opentelemetry-java-instrumentation
• telemetry.distro.version: 2.16.0
• opentelemetry-javaagent: 2.16.0-alpha
• Runtime: Vert.x (vertx-http-client)
• JVM: Eclipse Adoptium OpenJDK 21.0.9+10-LTS
• Framework: Vert.x con Http1xClientConnection
• OS: Linux 6.6.122+
• Arquitectura: amd64

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[laurit]

The HttpClientRequestTracingHandler appears in the stacktrace at line 40, between CombinedChannelDuplexHandler and HttpObjectEncoder, indicating the OTel instrumentation handler is part of the failing write chain.

Have a look at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/netty/netty-4.1/library/src/main/java/io/opentelemetry/instrumentation/netty/v4_1/internal/client/HttpClientRequestTracingHandler.java as you can see it just calls super.write. Whatever is the cause of your issue it is unlikely to be the HttpClientRequestTracingHandler

Have a look at eclipse-vertx/vert.x#5049 for a similar issue that doesn't involve otel agent.

[jvs87]

Thank you for the clarification and for pointing to the HttpClientRequestTracingHandler source code. After reviewing it, I can confirm that the handler only calls super.write when the message is not an HttpRequest, so it is indeed transparent to the pipeline and not the cause of the issue.
We also reviewed the Vert.x issue #5049 and it matches our scenario exactly — same IllegalStateException: unexpected message type: LastHttpContent$1, state: 0 stacktrace, same root cause (race condition in PipeImpl where LastHttpContent is written by the event loop thread before the request headers are enqueued by the non-event loop thread), and reproduced without OTel involved.
We will close this issue and redirect our investigation to the Vert.x side. We have already disabled the OTel agent as a temporary workaround (OTEL_JAVAAGENT_ENABLED=false) and confirmed the error stopped appearing.