Overload manager and load shed points

[maxbrunet]

Foreword

Started this as a discussion because I am not sure what the action item(s) would be (if any).

This is to continue with the good habit of opening an issue/discussion when something requires modify the Envoy config directly (with e.g. EnvoyPatchPolicy or EnvoyProxy.spec.bootstrap) to explore better support options due to the brittle nature of these hacks/workarounds.

Context

We have Envoy functioning as an L2 proxy receiving HTTP2 connections from the L1 proxy (downstream). The L1 proxy sits outside Kubernetes, and is not aware of Kubernetes pods or nodes, it only connects to the IP address of a network load-balancer. Envoy has an HorizontalPodAutoscaler configured to scale when memory usage reaches 70% (lower than the 80% threshold of the overload manager)

Problem

When traffic increases, downstream sends more requests/opens more HTTP2 connections, and the memory usage of Envoy increases, triggering its scale-out. However most connections are opened to the existing Envoy pods before the new pods are created. When the new pods come up, they are not used because from downstream's point of view, there are enough connections to serve the traffic, so the existing pods remain overloaded.

On top of rejecting requests (the overload action envoy.overload_actions.stop_accepting_requests), Envoy needs a way to communicate to downstream that the HTTP2 connection should be closed, so downstream would open a new one which should go to new pods. For that purpose exist the load shed points, specifically envoy.load_shed_points.http2_server_go_away_on_dispatch.

Current solution

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  bootstrap:
    type: Merge
    value: |
      overload_manager:
        loadshed_points:
          - name: "envoy.load_shed_points.http2_server_go_away_on_dispatch"
            triggers:
              - name: "envoy.resource_monitors.fixed_heap"
                threshold:
                  value: 0.98

Edit: the envoy.overload_actions.disable_http_keepalive action might be a better option (#8748 (comment))

Questions

• Should load shed points be configurable via a CR?
• Or should the overload manager as a whole be configurable? (Tweaking overload settings in gateway api? #3604)
• Or could configuring envoy.load_shed_points.http2_server_go_away_on_dispatch be a sane default to incorporate into the controller alongside the current actions?
• Or are we satisfied with the support offered by EnvoyProxy.spec.bootstrap?

[arkodg]

+1 to setting http2_server_go_away_on_dispatch by default with the same value as stop_accepting_requests
wdyt @envoyproxy/gateway-maintainers

[maxbrunet]

I wonder if we should also add the envoy.overload_actions.disable_http_keepalive action, it would be a more proactive approach to stop connection re-use when overloaded and would apply to HTTP/1 as well 🤔

[maxbrunet]

After experimenting further, this seems better to reduce memory usage by adding a new graceful action level before needing the more extreme measure of rejecting requests with errors. On sudden spikes where rejecting requests is inevitable, it reduces the time period during which the client sees errors (as opposed to using the same 0.98 threshold)

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  bootstrap:
    type: JSONPatch
    jsonPatches:
      - op: add
        path: /overload_manager/actions/-
        value:
          name: "envoy.overload_actions.disable_http_keepalive"
          triggers:
            - name: "envoy.resource_monitors.fixed_heap"
              threshold:
                value: 0.95

[arkodg]

also linking another runtime flag envoyproxy/envoy#41442