[Proposal] Supporting Endpoint Timeouts in the API Gateway

[O-sura]

Problem

The current API definition does not allow users to configure endpoint-level timeouts, which limits the gateway’s ability to enforce resiliency when upstream services are slow or unresponsive. Without explicit endpoint timeouts, requests may occupy gateway resources indefinitely, increasing the risk of cascading failures. As a result, we need a well-defined and intuitive configuration model that exposes endpoint timeouts while mapping cleanly to Envoy’s routing capabilities.

Proposed Solution

Introduce support for endpoint timeouts at both the API level and the operation level. At the API level, users can define an endpoint timeout that applies to all operations as a default. At the operation level, users can optionally override this value for specific endpoints that require different latency characteristics. For each operation, the gateway resolves a single effective timeout using the precedence operation-level → API-level → gateway default, and applies it to the corresponding Envoy route. A gateway-level default endpoint timeout is defined in the gateway configuration and acts as a safety net when no timeout is specified by the API author

API-level Endpoint Timeout (applies for all operations)

spec:
  requestTimeout: 2s

Operation-level Endpoint Timeout (override)

operations:
  - method: GET
    path: /{country_code}/{city}
    requestTimeout: 1s

  - method: GET
    path: /alerts/active
    requestTimeout: 5s

Benefits of proposed approach:

• Exposes endpoint timeout semantics clearly to API users
• Allows fine-grained control per endpoint while keeping sensible defaults
• Prevents indefinite waits and improves gateway resiliency
• Keeps the configuration extensible for future resiliency features

[BLasan]

Hi @O-sura ,

For upstream resiliency (backend), we need more granularity than simply configuring the requestTimeout. For example, we may need to configure circuit breakers, retries with backoff, etc. I would prefer introducing a separate object (or property) to configure all these resiliency-related parameters.

WDYT, @renuka-fernando?

Thanks!

[Krishanx92]

Let's proceed with upstream definition #369

[pubudu538]

We have other timeout values as well. Shall we make sure they are also captured here - https://apim.docs.wso2.com/en/4.2.0/deploy-and-publish/deploy-on-gateway/choreo-connect/endpoints/resiliency/timeout/

[renuka-fernando]

In envoy timeout is defined at the route level. In API YAML, if we configure this at the upstream level, this can lead to an issue when we are going to support the dynamic endpoint feature.

version: api-platform.wso2.com/v1alpha1
kind: HttpRestApi
metadata:
  name: weather-api
spec:
  displayName: Weather API
  version: v1.0 #
  context: /weather
  upstreamDefinitions:
    - name: my-upstream-1
      loadBalancer:
        strategy: ROUND_ROBIN
      timeout:
        connect: 5s
        request: 30s
        idle: 60s
      retry:
        attempts: 3
        conditions: [5xx, timeout, connection-failure]
        backoff: exponential
      upstreams: # Multiple upstreams
        - urls:
            - http://prod-backend-1:5000/api/v2
          weight: 80
        - urls:
            - http://sandbox-backend-1:5000/api/v2
          weight: 20

From a policy if the upstream my-upstream-1 is selected, there is no way to configure Envoy Route with the value defined here.

Envoy Route Action

{
  "cluster": ...,
  "cluster_header": ...,
  "weighted_clusters": {...},
  "cluster_specifier_plugin": ...,
  "inline_cluster_specifier_plugin": {...},
  "cluster_not_found_response_code": ...,
  "metadata_match": {...},
  "prefix_rewrite": ...,
  "regex_rewrite": {...},
  "path_rewrite_policy": {...},
  "path_rewrite": ...,
  "host_rewrite_literal": ...,
  "auto_host_rewrite": {...},
  "host_rewrite_header": ...,
  "host_rewrite_path_regex": {...},
  "host_rewrite": ...,
  "append_x_forwarded_host": ...,
  "timeout": {...}, ### TIMEOUT
  "idle_timeout": {...},
  "flush_timeout": {...},
  "early_data_policy": {...},
  "retry_policy": {...},
  "request_mirror_policies": [],
  "priority": ...,
  "rate_limits": [],
  "include_vh_rate_limits": {...},
  "hash_policy": [],
  "cors": {...},
  "max_grpc_timeout": {...},
  "grpc_timeout_offset": {...},
  "upgrade_configs": [],
  "internal_redirect_policy": {...},
  "internal_redirect_action": ...,
  "max_internal_redirects": {...},
  "hedge_policy": {...},
  "max_stream_duration": {...}
}

Suggestion

API Level and Route Level resilience field to define these values at the Envoy Route level. With this approach, we are keeping API Operation <-> Envoy Route mapping and API Upstream <-> Envoy Cluster.

resilience:
  timeout: 15s
  idle_timeout: 0s
  retry: {...}

[renuka-fernando]

cc: @nuwand, @pubudu538, @malinthaprasan, @Krishanx92

[renuka-fernando]

config.cluster.v3.Cluster

{
  "transport_socket_matches": [],
  "name": ...,
  "alt_stat_name": ...,
  "type": ...,
  "cluster_type": {...},
  "eds_cluster_config": {...},
  "connect_timeout": {...},
   ...
}

Connect timeout is a cluster-level config, so we can define this in API YAML, upstream config.

version: api-platform.wso2.com/v1alpha1
kind: HttpRestApi
metadata:
  name: weather-api
spec:
  displayName: Weather API
  version: v1.0
  context: /weather
  upstreamDefinitions:
    - name: my-upstream-1
      timeout:
        connect: 10s # Connect Timeout
      loadBalancer:
        strategy: ROUND_ROBIN
      upstreams:
        - urls:
            - http://prod-backend-1:5000/api/v2
          weight: 80
        - urls:
            - http://sandbox-backend-1:5000/api/v2
          weight: 20

[O-sura]

As discussed above,

• Approach followed in REST APIs: We introduced a new resilience block (with timeout and idleTimeout, as duration strings, unset falls back to the gateway's global route-timeout defaults) that maps to Envoy's RouteAction timeouts. This can be set at two levels: at the API level (applies to every operation) and at the operation/route level (overrides the API level). Precedence is most-specific-wins, per field — an operation's timeout overrides the API's timeout while still inheriting the API's idleTimeout if it doesn't set its own.

• For LLMProvider / LLMProxy: Introduce the same resilience block, but supported at the API level only(Since LLM routes are not user-authored).

Points to discuss:

• New yaml structure

LLMProvider

apiVersion: gateway.api-platform.wso2.com/v1
kind: LlmProvider
metadata:
  name: wso2-openai-provider
spec:
  displayName: OpenAI Provider
  version: v1.0
  template: openai
  context: /openai/latest
  upstream:
    url: https://api.openai.com/v1
    auth:
      type: api-key
      header: Authorization
      value: <API_KEY>
  resilience:
    timeout: 15s         # max time for the whole request→upstream-response on each route ("0s" disables)
    idleTimeout: 20s     # per-route stream idle timeout ("0s" disables); omit to use the gateway default
  accessControl:
    mode: deny_all
    exceptions:
      - path: /chat/completions
        methods:
          - POST
      - path: /models
        methods:
          - GET
      - path: /models/{modelId}
        methods:
          - GET

LLMProxy

apiVersion: gateway.api-platform.wso2.com/v1
kind: LlmProxy
metadata:
  name: openai-proxy
spec:
  displayName: OpenAI Proxy
  version: v1.0
  context: /openai-proxy
  provider:
    id: wso2-openai-provider
  resilience:
    timeout: 25s
    idleTimeout: 30s
  policies:
    - name: model-round-robin
      version: v1
      paths:
        - path: /chat/completions
          methods: ["*"]
          params:
            models:
              - model: gpt-4
              - model: gpt-4-turbo
              - model: gpt-3.5-turbo
            suspendDuration: 60
    - name: semantic-cache
      version: v1
      paths:
        - path: /chat/completions
          methods: ["*"]
          params:
            similarityThreshold: 0.85
            jsonPath: "$.messages[0].content"

• Are we going to support timeout in both proxy and provider level and document "set proxy.timeout ≥ provider.timeout," or is timeout-on-provider-only?

[O-sura]

cc: @nuwand @Krishanx92 @Arshardh @renuka-fernando

[O-sura]

Had a small discussion with @Arshardh and decided to move forward with the structure suggested above supporting it to be configured in both Provider and Proxy levels