What is the issue?
In some release after edge-25.2.1, linkerd-proxy began failing to forward gRPC status codes, error messages and trailers in cases where the total gRPC response trailer size was at or over roughly 12 kilobytes. The error message generated by the server is replaced by code 13, the message is replaced by stream terminated by RST_STREAM with error code: INTERNAL_ERROR, and no trailer information is forwarded.
This is tested on server version edge-26.4.2.
How can it be reproduced?
I have put together a full demonstration of this effect as a client-server combo and a helm chart:
https://github.com/odenio/echoerror
To install on your cluster, run:
helm upgrade --install echoerror ./chart/echoerror
To vary the amount of requested data in the trailers, add --set client.padMessageKb=<int>.
This is a toy server implementation that implements a single gRPC API, EchoError, which takes an EchoRequest message that lets the client specify the kind of result it wants:
message EchoRequest {
int32 code = 1;
string message = 2;
int32 pad_message_kb = 3;
}
The client code continuously sends messages to the server, iterating over codes 0 through 16, generating a random message per request, and setting a configured padding size. The response from the server is then compared to the request and any mismatch is logged.
On our cluster, if pad_message_kb is set to 11 or below, no mismatch errors are noted. But if it is set to 12 or above, all requests produce mismatches:
2026/04/22 15:31:49 MISMATCH TRAILER: no x-echo-message trailer received for code=16
2026/04/22 15:31:49 MISMATCH PAD: no x-echo-pad trailer received for code=16
2026/04/22 15:31:49 MISMATCH CODE: sent=16 (Unauthenticated) got=13 (Internal) message="test error code 16 at 2026-04-22T15:31:49.23084851Z"
2026/04/22 15:31:49 MISMATCH MESSAGE: sent="test error code 16 at 2026-04-22T15:31:49.23084851Z" got="stream terminated by RST_STREAM with error code: INTERNAL_ERROR" code=16
Logs, error output, etc
[ 158.165469s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[stream error received: unexpected internal error encountered]
[ 158.165506s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.4.26.79:49978: server: 10.4.26.68:4143: server 10.4.26.68:9090: service echoerror:9090: http2 error error.sources=[server 10.4.26.68:9090: service echoerror:9090: http2 error, http2 error, stream error received: unexpected internal error encountered]
[ 158.167073s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[stream error received: unexpected internal error encountered]
[ 158.167116s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.4.26.79:49978: server: 10.4.26.68:4143: server 10.4.26.68:9090: service echoerror:9090: http2 error error.sources=[server 10.4.26.68:9090: service echoerror:9090: http2 error, http2 error, stream error received: unexpected internal error encountered]
[ 158.168402s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[stream error received: unexpected internal error encountered]
[ 158.168451s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.4.26.79:49978: server: 10.4.26.68:4143: server 10.4.26.68:9090: service echoerror:9090: http2 error error.sources=[server 10.4.26.68:9090: service echoerror:9090: http2 error, http2 error, stream error received: unexpected internal error encountered]
[ 158.169892s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[stream error received: unexpected internal error encountered]
[ 158.169930s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.4.26.79:49978: server: 10.4.26.68:4143: server 10.4.26.68:9090: service echoerror:9090: http2 error error.sources=[server 10.4.26.68:9090: service echoerror:9090: http2 error, http2 error, stream error received: unexpected internal error encountered]
[ 158.171079s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[stream error received: unexpected internal error encountered]
[ 158.171121s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.4.26.79:49978: server: 10.4.26.68:4143: server 10.4.26.68:9090: service echoerror:9090: http2 error error.sources=[server 10.4.26.68:9090: service echoerror:9090: http2 error, http2 error, stream error received: unexpected internal error encountered]
[ 158.172517s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[stream error received: unexpected internal error encountered]
[ 158.172587s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.4.26.79:49978: server: 10.4.26.68:4143: server 10.4.26.68:9090: service echoerror:9090: http2 error error.sources=[server 10.4.26.68:9090: service echoerror:9090: http2 error, http2 error, stream error received: unexpected internal error encountered]
[ 158.173839s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[stream error received: unexpected internal error encountered]
[ 158.173877s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.4.26.79:49978: server: 10.4.26.68:4143: server 10.4.26.68:9090: service echoerror:9090: http2 error error.sources=[server 10.4.26.68:9090: service echoerror:9090: http2 error, http2 error, stream error received: unexpected internal error encountered]
[ 158.175163s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[stream error received: unexpected internal error encountered]
[ 158.175201s] INFO ThreadId(01) inbound:server{port=4143}:rescue{client.addr=10.4.26.79:49978}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.4.26.79:49978: server: 10.4.26.68:4143: server 10.4.26.68:9090: service echoerror:9090: http2 error error.sources=[server 10.4.26.68:9090: service echoerror:9090: http2 error, http2 error, stream error received: unexpected internal error encountered]
output of linkerd check -o short
linkerd-version
---------------
‼ cli is up-to-date
unsupported version channel: stable-2.14.10
see https://linkerd.io/2.14/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
is running version 26.4.2 but the latest edge version is 26.4.3
see https://linkerd.io/2.14/checks/#l5d-version-control for hints
‼ control plane and cli versions match
control plane running edge-26.4.2 but cli running stable-2.14.10
see https://linkerd.io/2.14/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-5c48449b76-pl8pd (edge-26.4.2)
* linkerd-destination-5c48449b76-qg6pc (edge-26.4.2)
* linkerd-destination-5c48449b76-s48x8 (edge-26.4.2)
* linkerd-identity-649c455559-4xbgf (edge-26.4.2)
* linkerd-identity-649c455559-m6klc (edge-26.4.2)
* linkerd-identity-649c455559-ng57w (edge-26.4.2)
* linkerd-proxy-injector-59f54fd677-6f9jc (edge-26.4.2)
* linkerd-proxy-injector-59f54fd677-g5s66 (edge-26.4.2)
* linkerd-proxy-injector-59f54fd677-q246s (edge-26.4.2)
see https://linkerd.io/2.14/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
linkerd-destination-5c48449b76-pl8pd running edge-26.4.2 but cli running stable-2.14.10
see https://linkerd.io/2.14/checks/#l5d-cp-proxy-cli-version for hints
linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
see https://linkerd.io/2.14/checks/#l5d-injection-disabled for hints
linkerd-viz
-----------
‼ viz extension proxies are up-to-date
some proxies are not running the current version:
* grafana-65db7cc7d9-25v8q (edge-25.11.3)
* metrics-api-5c4b8db764-n2dqj (edge-26.4.2)
* tap-847dc9cd6b-6mm5t (edge-26.4.2)
* tap-injector-7f445877b9-rmxqm (edge-26.4.2)
* web-7f4b65b674-wfdj6 (edge-26.4.2)
see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
grafana-65db7cc7d9-25v8q running edge-25.11.3 but cli running stable-2.14.10
see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cli-version for hints
Status check results are √
Environment
Kubernetes version 1.33.8-gke.1026000 on GKE
Possible solution
No response
Additional context
No response
Would you like to work on fixing this bug?
None
What is the issue?
In some release after edge-25.2.1, linkerd-proxy began failing to forward gRPC status codes, error messages and trailers in cases where the total gRPC response trailer size was at or over roughly 12 kilobytes. The error message generated by the server is replaced by code 13, the message is replaced by
stream terminated by RST_STREAM with error code: INTERNAL_ERROR, and no trailer information is forwarded.This is tested on server version edge-26.4.2.
How can it be reproduced?
I have put together a full demonstration of this effect as a client-server combo and a helm chart:
https://github.com/odenio/echoerror
To install on your cluster, run:
To vary the amount of requested data in the trailers, add
--set client.padMessageKb=<int>.This is a toy server implementation that implements a single gRPC API,
EchoError, which takes anEchoRequestmessage that lets the client specify the kind of result it wants:The client code continuously sends messages to the server, iterating over codes
0through16, generating a randommessageper request, and setting a configured padding size. The response from the server is then compared to the request and any mismatch is logged.On our cluster, if pad_message_kb is set to
11or below, no mismatch errors are noted. But if it is set to 12 or above, all requests produce mismatches:Logs, error output, etc
output of
linkerd check -o shortEnvironment
Kubernetes version 1.33.8-gke.1026000 on GKE
Possible solution
No response
Additional context
No response
Would you like to work on fixing this bug?
None