HAProxy runtime backends drift between ingress replicas after application deployment

[asjonos]

Description
We are seeing inconsistent live backend state between HAProxy Kubernetes Ingress replicas after redeploying an application behind a Service.

The generated haproxy.cfg on disk is updated correctly on all ingress pods after the redeploy, but the running HAProxy backend state is not updated on all pods. Some ingress replicas keep stale backend pod IPs in the live runtime state, mark them L4TOUT, and return 503. Other ingress replicas have the correct backend list and work normally.

As a workaround, adding:

haproxy.org/standalone-backend: "true"

to the Ingress makes the live backend update correctly after redeploy, and the problem disappears in our testing.

Environment

• haproxytech/kubernetes-ingress chart version: 1.49.0
• AKS 1.34.2
• Argo CD

HAProxy Kubernetes Ingress values

destNamespace: haproxy-ingress
appName: haproxy-ingress
haproxy-ingress:
  controller:
    ingressClassResource:
      name: haproxy-ingress
    ingressClass: haproxy-ingress
    replicaCount: 4
    service:
      type: LoadBalancer
      externalTrafficPolicy: Local
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
        service.beta.kubernetes.io/azure-load-balancer-resource-group: redacted
        service.beta.kubernetes.io/azure-load-balancer-ipv4: redacted
    serviceMonitor:
      enabled: true
    config:
      cr-defaults: haproxy-ingress/haproxy-defaults
      cr-frontend-http: haproxy-ingress/haproxy-frontend-http
      cr-frontend-https: haproxy-ingress/haproxy-frontend-https
      global-config-snippet: |
        ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 no-tls-tickets
        ssl-default-bind-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384
        ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
      syslog-server: "address:stdout, length:8192, format:raw, facility:daemon"
    extraArgs:
      - --default-backend-service=x/xxx
      - --disable-quic
---
apiVersion: ingress.v3.haproxy.org/v3
kind: Defaults
metadata:
  name: haproxy-defaults
  namespace: haproxy-ingress
spec:
  compression:
    direction: response
    minsize_res: 500
    algorithms:
      - gzip
    types:
      - text/plain
      - text/css
      - application/json
      - application/javascript
      - application/xml
      - text/xml
      - text/html
  redispatch:
    enabled: enabled
    interval: 1
  retries: 3
---
apiVersion: ingress.v3.haproxy.org/v3
kind: Frontend
metadata:
  name: haproxy-frontend-https
  namespace: haproxy-ingress
spec:
  name: haproxy-frontend-https
  http_response_rule_list:
    - type: del-header
      hdr_name: server
    - type: set-header
      hdr_name: Strict-Transport-Security
      hdr_format: max-age=31536000;\ includeSubDomains;\ preload;
  http_request_rule_list:
    - type: del-header
      hdr_name: X-Forwarded-For
    - type: set-header
      hdr_name: X-Forwarded-Host
      hdr_format: "%[req.hdr(Host)]"
    - type: set-header
      hdr_name: X-Forwarded-Proto
      hdr_format: https
    - type: set-header
      hdr_name: X-Forwarded-Port
      hdr_format: "443"
---
apiVersion: ingress.v3.haproxy.org/v3
kind: Frontend
metadata:
  name: haproxy-frontend-http
  namespace: haproxy-ingress
spec:
  name: haproxy-frontend-http
  http_request_rule_list:
    - type: redirect
      redir_type: scheme
      redir_value: https
      redir_code: 301
      cond: unless
      cond_test: "{ ssl_fc }"

Observed problem

During or after redeploy of an application behind a Service, some HAProxy ingress replicas continue to use stale backend pod IPs in the live backend state.

Symptoms:

• some ingress IPs return 503
• other ingress IPs still work
• another ingress, ingress-nginx, exposing the same app on another IP continues to work
• stats show L4TOUT on stale backend entries on affected HAProxy replicas

We initially saw this as intermittent 503 during redeploys of multiple simultaneous deployments and sometimes after pod termination events.

Evidence that generated config is correct, but runtime state is wrong

Runtime/stats output differs between HAProxy replicas

From the stats endpoint on each HAProxy ingress pod for backend zz_svc_yy-vvv_http:

haproxy-ingress-7cc59c8d86-kkj78

SRV_1 10.244.7.22:8080 L4OK
SRV_2 10.244.1.8:8080 L4OK
SRV_3 10.244.7.23:8080 L4OK

haproxy-ingress-7cc59c8d86-vhqgk

SRV_2 10.244.1.8:8080 L4OK
SRV_3 10.244.7.23:8080 L4OK

haproxy-ingress-7cc59c8d86-bfsl5

SRV_2 10.244.7.22:8080 L4OK
SRV_3 10.244.0.36:8080 L4OUT
SRV_4 10.244.2.113:8080 L4OUT

haproxy-ingress-7cc59c8d86-pvb7p

SRV_2 10.244.2.114:8080 L4OUT
SRV_3 10.244.0.36:8080 L4OUT
SRV_4 10.244.2.113:8080 L4OUT

These are not aligned across replicas, even though they are the same backend.

On-disk config is aligned across replicas

On-disk haproxy.cfg on the same pods:

haproxy-ingress-7cc59c8d86-kkj78

backend zz_svc_yy-vvv_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  server SRV_1 10.244.7.22:8080 enabled
  server SRV_2 10.244.1.8:8080 enabled
  server SRV_3 10.244.7.23:8080 enabled
  server SRV_4 127.0.0.1:1 disabled

haproxy-ingress-7cc59c8d86-vhqgk

backend zz_svc_yy-vvv_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  server SRV_1 10.244.7.22:8080 enabled
  server SRV_2 10.244.1.8:8080 enabled
  server SRV_3 10.244.7.23:8080 enabled
  server SRV_4 127.0.0.1:1 disabled

haproxy-ingress-7cc59c8d86-bfsl5

backend zz_svc_yy-vvv_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  server SRV_1 10.244.1.8:8080 enabled
  server SRV_2 10.244.7.22:8080 enabled
  server SRV_3 10.244.7.23:8080 enabled
  server SRV_4 127.0.0.1:1 disabled

haproxy-ingress-7cc59c8d86-pvb7p

backend zz_svc_yy-vvv_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  server SRV_1 10.244.1.8:8080 enabled
  server SRV_2 10.244.7.22:8080 enabled
  server SRV_3 10.244.7.23:8080 enabled
  server SRV_4 127.0.0.1:1 disabled

So the file on disk converges, but the live backend state does not.

Before/after haproxy.cfg from one ingress pod

To demonstrate that the stale addresses are from the previous deployment:

Before redeploy of application

backend zz_svc_yy-vvv_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  server SRV_1 127.0.0.1:1 disabled
  server SRV_2 10.244.2.114:8080 enabled
  server SRV_3 10.244.0.36:8080 enabled
  server SRV_4 10.244.2.113:8080 enabled
  server SRV_5 127.0.0.1:1 disabled

After redeploy of application

backend zz_svc_yy-vvv_http from haproxytech
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check
  server SRV_1 10.244.1.8:8080 enabled
  server SRV_2 10.244.7.22:8080 enabled
  server SRV_3 10.244.7.23:8080 enabled
  server SRV_4 127.0.0.1:1 disabled

Local conditions

This ingress is running in production under steady load of roughly 200 requests/second and handles that normally without any problems.

Because of that, load may be part of the trigger even though the setup is generally stable at this traffic level.

We are seeing this while doing multiple simultaneous deployments. Maybe it could be related to #768, that also occur during 'mass pod restart'.

[hdurand0710]

Hi @asjonos ,

Thansk for reporting this issue.
I was able to reproduce this issue but only with the following setup: having a set of Ingresses pointing to the same Service:

• some of them having the annotation" haproxy.org/standalone-backend: "true"
• some of them NOT having this annotation.
  In this case, I can reproduce the issue for the Backends related to the Ingresses NOT having the annotation standalone-backend.

If I have all the Ingresses pointing to the same Service not having the annotation, or all having, I'm not able at this point to reproduce.

Can you confirm that it's your configuration (mix of Ingresses pointing to the same Service, some with haproxy.org/standalone-backend: "true" , some without)?

[asjonos]

Yes, this is parts of my configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: yy-vvv-haproxy
  namespace: zz
spec:
  ingressClassName: haproxy-ingress
  rules:
  - host: www.tld.com
    http:
      paths:
      - backend:
          service:
            name: yy-vvv-critical
            port:
              name: http
        path: /criticalpaths
        pathType: Prefix
      - backend:
          service:
            name: yy-vvv
            port:
              name: http
        path: /api
        pathType: Prefix
      - backend:
          service:
            name: webtxt-robots
            port:
              name: http
        path: /robots.txt
        pathType: Prefix
      - backend:
          service:
            name: webtxt-404
            port:
              name: http
        path: /api/metrics
        pathType: Prefix
  tls:
  - hosts:
    - www.tld.com
    secretName: tls-secret-www
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    haproxy.org/cookie-persistence: sticky-dash-hangfire
    haproxy.org/standalone-backend: "true"
    haproxy.org/allow-list: 1.1.1.1
  name: yy-vvv-hangfire-haproxy
  namespace: zz
spec:
  ingressClassName: haproxy-ingress
  rules:
    - host: hangfire.tld.com
      http:
        paths:
          - backend:
              service:
                name: yy-vvv
                port:
                  name: http
            path: /hangfire
            pathType: Prefix
          - backend:
              service:
                name: webtxt-robots
                port:
                  name: http
            path: /robots.txt
            pathType: Prefix
          - backend:
              service:
                name: webtxt-404
                port:
                  name: http
            path: /api/metrics
            pathType: Prefix
  tls:
    - hosts:
        - hangfire.tld.com
      secretName: tls-secret-hangfire-haproxy
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    haproxy.org/cookie-persistence: application2
    haproxy.org/standalone-backend: "true"
  name: application2-haproxy
  namespace: app2
spec:
  ingressClassName: haproxy-ingress
  rules:
  - host: another.tld.com
    http:
      paths:
      - backend:
          service:
            name: application2
            port:
              name: http
        path: /
        pathType: Prefix
      - backend:
          service:
            name: webtxt-robots
            port:
              name: http
        path: /robots.txt
        pathType: Prefix
      - backend:
          service:
            name: webtxt-404
            port:
              name: http
        path: /metrics
        pathType: Prefix
  tls:
  - hosts:
    - another.tld.com
    secretName: tls-secret-another

[hdurand0710]

@asjonos ,
Thanks for you quick feedback. I'm checking with the team if we allow to mix standalone and not standalone for the same service.
If yes, that's definitively an issue to fix.
If not, we should not allow it.
Keep you posted.

[oktalz]

@asjonos this seems like a bug, you are allowed to have a setup like you have, so we will fix it.

[mdecalf]

We are seeing this while doing multiple simultaneous deployments. Maybe it could be related to #768, that also occur during 'mass pod restart'.

Yes, very similar and still applicable. I have not been able to identify the root cause until now.