Support for Gateway API

[pl4nty]

Has there been any discussions on implementing support for the k8s beta Gateway API? The API should be flexible enough to implement most Tunnel capabilities, and uses built-in resources so new users can easily onboard/offboard (eg ingress2gateway).

Not sure whether this project would be the best fit, or a separate project from scratch. As I imagine this design as a gateway controller implementation, rather than an operator.

[adyanth]

I haven't explored the Gateway API too much myself, but the reason for converting the operator from targeting the ingress to the service was that you can do anything you could by pointing to the ingress and more with a service. Same thing goes for the Gateway API, at the end, all of them end in a service, which is where you can point the tunnel to directly, since the cloudflared is already a reverse proxy.

Unless you are suggesting to make an ingress/gateway controller out of cloudflared, for which, there are custom inputs that it needs, like credentials and which tunnel for each ingress that does not have place in the GatewayAPI CRD. We would need to fall back to annotations (or another CRD linked via the GatewayClass), which is what we moved away to custom CRDs. And instead of making the CRD+Ingress/HTTPRoute combo, we are just making it CRD+Service combo, reducing a layer of complexity, since there is no point of an Ingress if you anyway need to specify details in the CRD. You could still have Ingress/HTTPRoute, they will still work locally, but cloudflared will not target it. Same argument about pointing to the ingress for getting certain features like authentication, you can still point to the ingress/gateway controller's service to get all those features.

[pl4nty]

Yeah, I'm suggesting a gateway controller to allow configuring cloudflared with Gateway API resources. Definitely agree about avoiding annotations and additional CRDs where possible. GatewayClass spec.ParametersReference might work for some parameters, but Policy Attachment seems designed for this usecase, and adds some reusability that isn't available with the current CRDs.

Needing additional CRDs for routing should be avoidable. Here's a quick mapping that might be possible:

Current API	Gateway API
Install from yaml	Add GatewayClass to yaml
Tunnel/ClusterTunnel	Gateway, with namespace filtering via spec.listeners[].allowedRoutes
Tunnel/ClusterTunnel spec	New TunnelPolicy CRD
TunnelBinding HTTP(S)/TCP/UDP	HTTPRoute/TLSRoute/TCPRoute/UDPRoute
TunnelBinding SSH/RDP	TCPRoute spec.parentRefs.sectionName?

A new CRD (eg TunnelPolicy) would be required to handle credentials etc, but policy attachment might simplify this overall. A TunnelPolicy could be applied to multiple gateways, so config would be reusable between tunnels (I have a lot of duplicated config with the current ClusterTunnels/Tunnels).

Ongoing use of a gateway would only need the various routes, and those come with extra functionality. Eg HTTPRoutes could allow easier implementation of cloudflared's simple path-based routing.

TCP/UDP routes are still experimental though, and the API as a whole only reached beta recently. So maybe not something for the short term. I've just had great experiences with the API on other platforms, and it's worked really well for other flexible routing platforms.

[adyanth]

Ahh, so instead of a TunnelBinding, use one of the Gateway API's CRDs. That is a good way if everything that Cloudflare supports can be mapped neatly without abusing the Gateway API's meaning of the fields. I would have to look into this a bit more. Thanks for the idea! I will keep this as an enhancement for now.

[dudo]

Gateway classes support a lot more than what cloudflared needs to be concerned with. GAPI just creates a service of type LoadBalancer, which you can point the daemon/operator to.

This feature feels like it's asking too much of cloudflared.

[pl4nty]

@dudo GAPI doesn't require a LoadBalancer, and its core functionality seems like a good match for cloudflare-operator at first glance. particularly since cloudflare-operator originally used ingress annotations, and seemingly moved to CRDs for extensibility (#51) which is a GAPI design goal.

With Cloudflare adding more tunnel features (eg path-based routing), the existing CRDs might be difficult to maintain, vs GAPI where these concepts are standardised

[pl4nty]

I've been hitting too many reconciliation bugs with this operator recently, so I'm going to try implementing GAPI separately with remote tunnel config. aiming for tunnel/hostname/service/path reconciliation first, since originRequest/warp-routing and cloudflared args/env may need an experimental GAPI policy CRD. full conformance may not be possible anyway due to missing Cloudflare features (eg error handling with specific HTTP codes)

[pl4nty]

My Gateway controller has been stable for around 6 months. I haven't implemented policy attachment or non-HTTP protocols yet though

[cyclingwithelephants]

I spent a couple months looking at this for my own use, and after attending some gateway API office hours it seems the support for load balancing just isn't there. It's very doable for treading cloudflared as a more traditional reverse proxy (i.e. to support HTTPRoutes/TCPRoutes only, but there was no way to express using cloudflared as a L3 Load balancer, nor was there a desire to extend the gateway API in this way. It can be done, but I prefer to use ingress-nginx to offload SSO and other features and treat cloudflare tunnels as the load balancer only, as it makes the infrastructure definition more modular

I see tailscale have done a similar thing where they have a custom controller, which allows them to back tailscale with any gateway implementation they want

Additionally, the conformance tests gave some difficulties:

• the host in the conformance tests is hardcoded to example.com (I asked about parametrising it, they said no) and this means writing a mock for the cloudflare API, which seemed a similarly complex task to writing the controller itself
• conformance tests very reasonably require handling nested subdomains i.e. a.b.example.com but this isn't a free feature for cloudflare tunnels and cloudflare weren't sponsoring me :D

Because of these difficulties, my development on it stalled and I gave up. I'm curious how you overcame these @pl4nty ?

[pl4nty]

I'm not sure what you mean by cloudflared as a L3 load balancer, what sort of routes do you want to configure? I've only implemented HTTPRoute so far, but I've been meaning to rewrite with some boilerplate from Cilium like https://github.com/caddyserver/gateway. And add TLSRoute/TCPRoute along the way. The project started on a weekend to learn golang and has been rock-solid in my homelab, so I haven't had motivation to revisit it until recently.

My implementation is non-conformant due to Cloudflare routing limitations and lack of interest on my part, but I hope to borrow a lot of validation code from Cilium. The tests work ok though - the Tunnels API accepts invalid routes like example.com just fine, and it looks like the DNS API treats example.com as a no-op.

[tszyszko]

@cyclingwithelephants did you have any further luck with cloudflared as a gateway controller / exposing a loadbalancer service to front it with envoy?

[tszyszko]

An ideal setup would allow it to be used like the tailscale operator, i.e:

  kind: GatewayClass
  metadata:
  name: tailscale
  spec:
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: tailscale-proxy
      namespace: envoy
  ---
  apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: EnvoyProxy
  metadata:
    name: tailscale-proxy
  spec:
    provider:
      type: Kubernetes
      kubernetes:
        envoyDeployment:
          replicas: 3
        envoyService:
          type: LoadBalancer
          loadBalancerClass: tailscale
          annotations:
            tailscale.com/proxy-group: ingress-proxies

[cyclingwithelephants]

@pl4nty I realised I got the layer wrong in my previous message, I meant L4. What I was really hoping to express was support for arbitrary transport-layer protocols, rather than being limited specifically to TCPRoute and UDPRoute resources. Conceptually, I wanted to be able to express:

“Everything that arrives at gateway X should be forwarded to gateway Y.”

A good example is an AWS NLB, which operates at L4. That’s exactly the kind of behaviour I was hoping to model. With a more general L4 route abstraction, you could support protocols beyond TCP and UDP e.g. QUIC, ICMP, or custom database protocols such as those used by Cassandra or CockroachDB.

That said, Cloudflare Tunnels don’t support arbitrary transport protocols anyway. So in practice, I think the Gateway API is expressive enough for this use case — even if it wouldn’t be sufficient for something like MetalLB or an AWS NLB.

---

@tszyszko I’ve had good results using the current cloudflare-operator in front of ingress-nginx. Actually I documented how I do this in my lab. It hasn’t been updated for the Gateway API, but the main change would just be that the tunnelBinding points to the Service created by the Gateway resource for the reverse proxy. To me, this highlights the strength of separating concerns between the load balancer implementation and the reverse proxy implementation.

I’ve experimented a bit and confirmed that chaining gateways is possible. So this project could implement TCPRoute and UDPRoute resources and, in theory, satisfy the load balancer abstraction that way.

I haven’t experimented with Envoy yet. What you’re raising is interesting, though, because Tailscale implements a LoadBalancer controller and doesn’t require any knowledge of the Gateway API AFAIK. It’s possible that this is actually the right abstraction layer for what we’re discussing here although it’s not what’s being requested in this case.