[serve] Enable HAProxy redispatch and retry-on for backend resilience

[harshit-anyscale]

Summary

Add option redispatch and retry-on conn-failure empty-response to each generated HAProxy backend block (python/ray/serve/_private/haproxy_templates.py). This makes the existing default retries 3 actually pick a different server on retry, and extends retry coverage to the safe failure modes (connect failure, empty response).

Motivation

During load testing of a multi-app Serve service with HAProxy ingress, we observed a tail of HTTP 5xx returned to clients despite zero 5xx coming from upstream replicas. The HAProxy Prometheus metrics showed:

• haproxy_backend_http_responses_total{code="5xx"} ≈ 0 (replicas weren't returning 5xx)
• haproxy_frontend_http_responses_total{code="5xx"} > 0 (clients saw 5xx)

Root cause: when a replica enters the window between dying and being marked DOWN by HAProxy's health check, requests still get dispatched to it. HAProxy's default retries 3 retries against the same dead server (no redispatch), exhausts retries, falls through to the SERVE_PROXY_ACTOR-fallback backup, and ultimately surfaces a synthesized 502 (which the existing errorfile 502 → 500.http config maps to a 500 visible to the client).

option redispatch makes the retries pick a different healthy server, eliminating most of this tail. retry-on conn-failure empty-response extends retry coverage from "connect-only" (default) to also handle the case where the connection succeeded but the server died before responding.

Why these specific options

Directive	Included?	Reason
option redispatch	✅	Converts default retries 3 from useless-on-same-dead-server to actually useful.
retry-on conn-failure	✅	Connect refused / RST during handshake — request never reached server, safe to retry.
retry-on empty-response	✅	Connection accepted but server closed before responding — safe to retry, no response bytes seen by client.

The two retry conditions chosen are the body-stream-safe subset that don't risk double-execution of side-effecting application logic.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request updates the HAProxy configuration templates and associated tests to include "option redispatch" and "retry-on conn-failure empty-response" for improved reliability. The review feedback suggests explicitly defining the retry count and adding a note regarding the HAProxy 2.0+ version requirement for the "retry-on" directive to ensure environment compatibility and consistent behavior.

I am having trouble creating individual review comments. Click here to see my feedback.

python/ray/serve/_private/haproxy_templates.py (94-100)

The retry-on directive was introduced in HAProxy 2.0. If the environment uses an older version of HAProxy (e.g., stock versions on older LTS distributions), this configuration will cause HAProxy to fail at startup.

Additionally, while HAProxy defaults to retries 3, it is more robust to set this explicitly in the configuration to ensure consistent behavior across different environments and custom HAProxy builds.

    # On a server failure, retry on a different server (combined with the
    # explicit `retries 3`) instead of giving up to the backup or returning
    # a synthesized 5xx. retry-on covers connection-establishment failures
    # and "connected but no response yet" cases — both are body-stream-safe
    # and don't risk double-execution.
    # Note: retry-on requires HAProxy 2.0+.
    option redispatch
    retries 3
    retry-on conn-failure empty-response

python/ray/serve/tests/test_haproxy_api.py (329-335)

Updating the test fixture to match the suggested template changes (explicit retries 3 and version note).

    # On a server failure, retry on a different server (combined with the
    # explicit `retries 3`) instead of giving up to the backup or returning
    # a synthesized 5xx. retry-on covers connection-establishment failures
    # and "connected but no response yet" cases — both are body-stream-safe
    # and don't risk double-execution.
    # Note: retry-on requires HAProxy 2.0+.
    option redispatch
    retries 3
    retry-on conn-failure empty-response

python/ray/serve/tests/test_haproxy_api.py (354-360)

Updating the test fixture to match the suggested template changes (explicit retries 3 and version note).

    # On a server failure, retry on a different server (combined with the
    # explicit `retries 3`) instead of giving up to the backup or returning
    # a synthesized 5xx. retry-on covers connection-establishment failures
    # and "connected but no response yet" cases — both are body-stream-safe
    # and don't risk double-execution.
    # Note: retry-on requires HAProxy 2.0+.
    option redispatch
    retries 3
    retry-on conn-failure empty-response

[cursor]

Notice-level marker may defeat incremental optimization entirely

Medium Severity

Including "[notice]" in error_markers within _send_runtime_command may cause the method to raise RuntimeError on successful HAProxy operations. HAProxy's CLI can return [NOTICE]-level messages for non-error events (e.g., server state transitions after enable server, or registration confirmations after add server in some versions). If the HAProxy version in use returns [NOTICE] on any successful runtime command, the incremental path in try_apply_servers_dynamically will always exception out and fall back to a full reload, completely negating the performance optimization that this PR introduces.

 

^{Reviewed by Cursor Bugbot for commit efbe04e. Configure here.}

[cursor]

Failed reload never retried due to premature state update

Medium Severity

try_apply_servers_dynamically calls set_backend_configs(new_backend_configs) on every code path before returning, including when it returns False for structural changes. In _apply_backend_update, if the subsequent reload() call then raises an exception, self.backend_configs already reflects the new desired state. On the next broadcast with identical target groups, _compute_backend_diff sees no difference and try_apply_servers_dynamically returns True (no-op), so the reload is never retried. HAProxy permanently serves stale routing until a different config change arrives. The old code unconditionally regenerated and reloaded on every broadcast, ensuring convergence after transient failures.

Additional Locations (1)

• python/ray/serve/_private/haproxy.py#L1562-L1584

 

^{Reviewed by Cursor Bugbot for commit ce527e4. Configure here.}

[cursor]

Coalescer silently drops pending broadcast on apply failure

Low Severity

When _apply_backend_update raises inside _coalesce_and_apply, the except block logs the error and exits. If a broadcast arrived during the failed apply (setting _coalesce_pending = True), the task ends without re-checking the flag. No new task is scheduled until the next broadcast arrives. If no further broadcasts come, the pending update is silently lost and HAProxy stays stale indefinitely.

 

^{Reviewed by Cursor Bugbot for commit ce527e4. Configure here.}

[cursor]

Health probe hardcodes 127.0.0.1 ignoring configured bind address

Medium Severity

_probe_http_listener always connects to "127.0.0.1", but HAProxy's frontend_host property maps a non-loopback http_options.host (e.g., "192.168.1.5") directly into the bind directive — meaning HAProxy only listens on that specific interface. In that configuration the probe fails with connection refused on every health check, causing the controller to consider the proxy unhealthy and kill/recreate it in a loop, even though HAProxy is perfectly functional on its configured address.

 

^{Reviewed by Cursor Bugbot for commit aded8a7. Configure here.}

[cursor]

Duplicated error markers tuple across two methods

Low Severity

The error_markers tuple is defined identically inside both _send_runtime_command and _send_runtime_commands. If a marker is later added to one but not the other, error detection would silently diverge between single-command and batched-command paths. Additionally, _send_runtime_command (singular) appears to be unused — only _send_runtime_commands (plural) is called from _rebind_slot_pools and try_apply_servers_dynamically.

Additional Locations (1)

• python/ray/serve/_private/haproxy.py#L1590-L1605

 

^{Reviewed by Cursor Bugbot for commit 9dbe27e. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 6 total unresolved issues (including 5 from previous reviews).

^{Reviewed by Cursor Bugbot for commit 8bb9997. Configure here.}

[cursor]

Test expects wrong hard-stop-after value (600 vs 1800)

Medium Severity

The test's HAProxyConfig stub does not explicitly set hard_stop_after_s, so it inherits the default RAY_SERVE_HAPROXY_HARD_STOP_AFTER_S. This constant was changed from 120 to 1800 in this PR, but the expected config string says hard-stop-after 600s. The actual rendered value will be 1800s, causing the full-config comparison assertion to fail.

 

^{Reviewed by Cursor Bugbot for commit 8bb9997. Configure here.}