Consider switching from Nginx to Traefik

[lunika]

Feature Request

This issue is a reminder to explore this possibility, not a hard requirement.

Is your feature request related to a problem or unsupported use case? Please describe.

We are having trouble scaling WebSockets using Nginx. Nginx is not easy to configure with WebSocket. And we have a hard requirement with the usage of the Nginx module ngx_http_auth_request_module.

Describe the solution you'd like

Traefik managed WebSocket natively, The auth_request module can be replaced by the Traefik ForwardAuth middleware.

In Kubernetes, we have to investigate between the Traefik ingress and the Kubernetes CRD installation. The CRD seems to be more configurable and probably have no downtime when the configuration is changed.

[pierreozoux]

I guess you envision using traefik for ingress. (Traefik can also be used as gateway api).

I think there is a consensus in the kubernetes community that Kubernetes ingress is over, and the future is toward gateway api.

The current frustration is just that gateway api ecosystem is just not mature enough.

I did an investigation in december around this topic, for us, lasuite: https://docs.numerique.gouv.fr/docs/8ccae95d-77b4-4237-9c76-5c0cadd5067e/

But to go back to our problem websocket burst when nginx controller reloads.

I think there would be several options:

1. have your own nginx ingress controller, so that when there are updates in the cluster, you are not impacted, you are only impacted when you do update your scope (annsi will like it, we can have ip filtering from the waf, a lot of nice features!)
2. switch from nginx to traffic, basically, you'd get the same benefits from above, but with a different data plane (maybe with better features, but maybe worse things, it is new, so there are new things to consider)
3. switch entirely to gateway api - here there are a lot of innovation token to investigate :) it is not a small project, it is a medium project!
4. use gateway api only for websocket

The 1 would be a no brainer as the least amount of work, complexity.
The 4 would be my preferred option, it would require a bit more effort, but it would set us on the road to gateway api.