[Feat][Router]: Add automatic retry with exponential backoff and jitter

[ikaadil]

Overview

Implements automatic retry for transient HTTP failures with configurable exponential backoff and jitter. This enhances the router's resilience by automatically retrying failed requests instead of immediately returning errors to clients.

Details: https://medium.com/@avnein4988/mitigating-the-thundering-herd-problem-exponential-backoff-with-jitter-b507cdf90d62

Changes

Core Implementation

• RetryConfig dataclass: Configurable retry parameters with exponential backoff calculation
• Retry logic: Integrated into request processing flow in route_general_request()
• Retryable status detection: Function to identify transient failures

CLI Arguments

Added retry configuration options:

• --retry-max-retries (default: 5)
• --retry-initial-backoff-ms (default: 50)
• --retry-max-backoff-ms (default: 30000)
• --retry-backoff-multiplier (default: 1.5)
• --retry-jitter-factor (default: 0.2)
• --disable-retries: Disable all retries

Key Features

Retryable Status Codes

Automatically retries on transient failures:

• 408 - Request Timeout
• 429 - Too Many Requests
• 500 - Internal Server Error
• 502 - Bad Gateway
• 503 - Service Unavailable
• 504 - Gateway Timeout

Exponential Backoff with Jitter

Prevents thundering herd through randomized delays:
Formula: delay = min(initial_backoff_ms × (multiplier ^ attempt), max_backoff_ms)
With jitter: D' = D × (1 + U[-j, +j])
Example with defaults:

• Attempt 0: ~50ms
• Attempt 1: ~75ms
• Attempt 2: ~112ms
• Attempt 3: ~168ms
• Attempt 4: ~253ms

Backward Compatibility

• Removed existing max_instance_failover_reroute_attempts behavior
• Falls back to single attempt when set to 0
• No breaking changes to existing functionality

Usage

vllm-router --port 8000 \
    --service-discovery static \
    --static-backends "http://localhost:9001,http://localhost:9002" \
    --static-models "facebook/opt-125m,facebook/opt-125m" \
    --routing-logic roundrobin \
    --retry-max-retries 5 \
    --retry-initial-backoff-ms 100 \
    --retry-max-backoff-ms 60000 \
    --retry-backoff-multiplier 2.0 \
    --retry-jitter-factor 0.1

[gemini-code-assist]

Code Review

This pull request introduces a retry mechanism with exponential backoff and jitter for transient failures, aligning the router's behavior with the sglang model gateway. The changes include a new RetryConfig dataclass, CLI arguments for configuration, updated documentation, and logic in the request service to handle retryable HTTP status codes (408, 429, 500, 502, 503, 504). Feedback identifies a logic error where retries are effectively disabled by default due to the max_attempts calculation, the inclusion of an unused last_response variable, and a concern that blacklisting URLs for transient errors prevents retrying the same backend in single-node environments.

[ikaadil]

@ruizhang0101 could you please review the MR? Thanks!

[ruizhang0101]

@aeon-x Could you take a look at this?

[aeon-x]

Hey @ikaadil, i think retrying should be an optional, as most users would need fast fail over.

Can you make sure that this retry mechanism is turned off unless it is explictly turned on by a flag?

[ikaadil]

Can you make sure that this retry mechanism is turned off unless it is explictly turned on by a flag?

Done