Skip to content

port-forward uses a SPDY-only dialer and fails behind proxies that do not support SPDY #3258

Description

@gmolau

Bug report

General Information

  • Cilium CLI version: v0.16.24 (the code below is unchanged on main and in v0.19.4)
  • Kubernetes: kubectl v1.31+, any apiserver version
  • Platform: environment-independent — reproduces whenever an HTTP proxy / gateway /
    load balancer that does not support SPDY sits in front of the kube-apiserver

Problem

cilium port-forward (e.g. cilium hubble port-forward) fails with
error upgrading connection: ... Upgrade request required when an HTTP proxy that
does not support SPDY is in front of the kube-apiserver.

The dialer in pkg/k8s/portforward/portforward.go (vendored from cilium/cilium)
is SPDY-only, with no WebSocket attempt and no fallback:

roundTripper, upgrader, err := spdy.RoundTripperFor(pf.config)
dialer := spdy.NewDialer(upgrader, &http.Client{Transport: roundTripper}, http.MethodPost, req.URL())

SPDY has been deprecated since 2015 (KEP-4006) and is no longer supported by many
proxies/gateways/load balancers. kubectl made WebSocket the default for port-forward
in v1.31, with SPDY as a fallback.

Why exec is unaffected

The exec path already uses the apimachinery WebSocket/SPDY fallback dialer
(createDialer, added in #37538). port-forward was never migrated.

#38988 later set exec to SPDY-primary with WebSocket fallback, due to hardcoded
idle-timeout issues in the upstream remotecommand WebSocket executor. That is
specific to remotecommand and does not apply to port-forward, which uses a separate
path (portforward.NewSPDYOverWebsocketDialer / the tunneling dialer).

Fix

Use the apimachinery fallback dialer in portforward.go with WebSocket as primary
and SPDY as fallback, matching kubectl's port-forward behaviour since v1.31:

spdyRoundTripper, upgrader, err := spdy.RoundTripperFor(pf.config)
if err != nil {
    return nil, err
}
spdyDialer := spdy.NewDialer(upgrader, &http.Client{Transport: spdyRoundTripper}, http.MethodPost, req.URL())

tunnelingDialer, err := portforward.NewSPDYOverWebsocketDialer(req.URL(), pf.config)
if err != nil {
    return nil, err
}

dialer := portforward.NewFallbackDialer(tunnelingDialer, spdyDialer, func(err error) bool {
    return httpstream.IsUpgradeFailure(err) || httpstream.IsHTTPSProxyError(err)
})

SPDY remains an automatic fallback for older API servers, so the change is
backward-compatible.

How to reproduce

  1. Place an HTTP proxy that does not support SPDY in front of the kube-apiserver.
  2. Run cilium hubble port-forward (or any cilium command that port-forwards).
  3. The SPDY upgrade is rejected and the command fails.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions