Readiness probe couples to a non-spec data-plane topology, breaking conformant off-cluster / tunnel / serverless implementations

[lexfrei]

Summary

net-gateway-api gates KIngress readiness on an HTTP probe dialed directly at the gateway data plane. That hard-requires the data plane to be a set of directly-probeable in-cluster HTTP endpoints — a topology the Gateway API spec does not define and does not require. Coupling readiness to an assumption outside the Gateway API contract is a bug, not a limitation of the implementations it breaks: a fully spec-conformant Gateway whose data plane is off-cluster (tunnel, serverless, edge) is left permanently Uninitialized even though every signal the Gateway API contract exposes is green.

This is the implementation-specific coupling raised in #663, which auto-closed as stale (NOT_PLANNED) without resolution. Filing standalone so the defect stays visible.

Why this is net-gateway-api's bug, not the implementation's

I maintain a Gateway API controller whose data plane is a Cloudflare Tunnel (cloudflared, outbound-only). There is no in-cluster L7 Service to dial, and the Gateway advertises a Hostname-type status.address (the tunnel CNAME) — a valid AddressType per the Gateway API spec. The generated HTTPRoutes are Accepted=True / ResolvedRefs=True, the Gateway is Programmed=True, and traffic serves end-to-end through the edge. The only failure is the readiness probe, because it dials status.addresses[0] (a tunnel CNAME) which does not resolve in-cluster. Nothing the spec requires is missing on my side — the probe assumes something the spec never promised.

The narrow defect

I am not arguing "just trust Programmed." It is defined as config "assumed to be ready soon," not "serving now," so I understand wanting positive end-to-end confirmation before shifting traffic — Gateway API has no "this config is serving" signal. The bug is narrower: the probe makes direct in-cluster data-plane reachability a hard precondition with no escape hatch, so it silently excludes conformant implementations instead of degrading to the contract's own readiness signal.

What would fix it

• A capability / opt-out letting a Gateway implementation declare its data plane non-probeable, with net-gateway-api falling back to Gateway API status (Programmed + matching observedGeneration) — the implementation then owns reporting readiness honestly; and/or
• if probing stays mandatory, document the in-cluster-reachability requirement explicitly, so a non-probeable data plane is a known-unsupported case rather than a silent permanent Uninitialized.

Reproduction (kind · Knative Serving v1.22.1 · net-gateway-api v1.22.1)

config-gateway points at a Gateway backed by the tunnel controller, off-cluster mode (no service:, so the probe targets the Gateway status address):

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-gateway
  namespace: knative-serving
data:
  external-gateways: |
    - class: cloudflare-tunnel
      gateway: knative-serving/knative-gateway
      supported-features:
      - HTTPRouteRequestTimeout
  local-gateways: |
    - class: cloudflare-tunnel
      gateway: knative-serving/knative-gateway
      supported-features:
      - HTTPRouteRequestTimeout

Deploy any ksvc (e.g. helloworld-go). Everything the Gateway API contract exposes is green:

# HTTPRoute (generated by net-gateway-api)
Accepted=True (Accepted)
ResolvedRefs=True (ResolvedRefs)

# Gateway
Accepted=True / Programmed=True
status.addresses: [{type: Hostname, value: <tunnel-id>.cfargotunnel.com}]

# revision pod
helloworld-00001-deployment-...  2/2  Running

# KIngress
NetworkConfigured=True
LoadBalancerReady=Unknown (Uninitialized: Waiting for load balancer to be ready)
Ready=Unknown (Uninitialized)

# ksvc
READY=Unknown  REASON=Uninitialized   # permanently

…but the prober can never reach the data plane, on a loop:

Probing of http://helloworld.default.example.com/ failed,
IP: <tunnel-id>.cfargotunnel.com:80, ready: false,
error: dial tcp: lookup <tunnel-id>.cfargotunnel.com on 10.96.0.10:53: no such host

The HTTPRoute config itself is fine — weighted backendRefs, the RequestHeaderModifier filters (including the injected K-Network-Hash), and the exact header matches are all accepted and programmed. Only the probe fails.

The K-Network-Hash contract itself is satisfiable by a non-Envoy data plane (PoC)

Four changes together make the probe pass; each was individually necessary:

1. The data-plane proxy runs in "tunnel mode" where traffic flows in-process from cloudflared and nothing listens in-cluster. I added an in-cluster HTTP listener on the proxy port serving the same L7 handler, alongside the tunnel (both coexist — the edge stays connected).
2. A Service whose first declared port is the proxy port — net-gateway-api dials the Service's first port, so a multi-port Service with the config-API port first is probed on the wrong port.
3. A NetworkPolicy rule allowing the proxy port from the net-gateway-api controller's namespace (the data-plane policy otherwise only allows its config-API port from the controller namespace, so the probe times out).
4. config-gateway service: pointed at that Service.

With all four, an in-cluster probe returns exactly what the verifier wants:

$ curl -s -D - -o /dev/null \
    -H "Host: helloworld.default.example.com" \
    -H "K-Network-Probe: probe" -H "K-Network-Hash: override" \
    http://<proxy-pod-ip>:8080/
HTTP/1.1 200 OK
K-Network-Hash: 140a84b099e0d1506f815c27abf87326679e7af503c8b4ff6aad367d653fd88d

KIngress goes Ready=True, the ksvc goes Ready, and publishing a new revision rolls over cleanly (the proxy then serves the new revision's content). So the K-Network-Hash contract is fully satisfiable by a non-Envoy data plane — the blocker is purely that the probe must reach an in-cluster HTTP listener, which an off-cluster/tunnel data plane has no reason to expose.

Prior discussion: #663.