Skip to content

Set TIMEOUT and CONNECTTIMEOUT on pycurl handles in CurlClient#2488

Open
jlmorton wants to merge 3 commits into
celery:mainfrom
jlmorton:fix/curl-client-missing-timeouts
Open

Set TIMEOUT and CONNECTTIMEOUT on pycurl handles in CurlClient#2488
jlmorton wants to merge 3 commits into
celery:mainfrom
jlmorton:fix/curl-client-missing-timeouts

Conversation

@jlmorton
Copy link
Copy Markdown

The CurlClient._setup_request() method configures many pycurl options but never sets TIMEOUT or CONNECTTIMEOUT on the curl handle. The Request class already defines request_timeout (30s) and connect_timeout (30s), but these values are never applied to the underlying curl transfer.

Without TIMEOUT, pycurl will wait indefinitely for a response on a socket that remains open but stops delivering data. This happens during network partitions, proxy restarts, or sidecar terminations — the TCP connection stays established (no RST, no FIN) but the remote end stops sending. The in-flight HTTP request hangs forever, which permanently breaks the SQS async polling loop: the promise callback from _get_from_sqs never fires, so _loop1 is never re-invoked and the consumer stops polling for messages. The worker process stays alive (event loop keeps running, liveness probes pass) but never consumes another message.

With TIMEOUT set, pycurl aborts the stalled transfer after request_timeout seconds and returns an error. The existing error handling in _process/_process_pending_requests picks this up, the transport reconnects, and the polling loop resumes normally.

The CurlClient._setup_request() method configures many pycurl options
but never sets TIMEOUT or CONNECTTIMEOUT on the curl handle. The
Request class already defines request_timeout (30s) and
connect_timeout (30s), but these values are never applied to the
underlying curl transfer.

Without TIMEOUT, pycurl will wait indefinitely for a response on a
socket that remains open but stops delivering data. This happens during
network partitions, proxy restarts, or sidecar terminations — the TCP
connection stays established (no RST, no FIN) but the remote end stops
sending. The in-flight HTTP request hangs forever, which permanently
breaks the SQS async polling loop: the promise callback from
_get_from_sqs never fires, so _loop1 is never re-invoked and the
consumer stops polling for messages. The worker process stays alive
(event loop keeps running, liveness probes pass) but never consumes
another message.

With TIMEOUT set, pycurl aborts the stalled transfer after
request_timeout seconds and returns an error. The existing error
handling in _process/_process_pending_requests picks this up, the
transport reconnects, and the polling loop resumes normally.
@auvipy auvipy requested review from auvipy and Copilot and removed request for Copilot March 12, 2026 12:01
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds explicit per-request timeout configuration to the pycurl-based async HTTP client to prevent indefinitely hanging sockets (notably impacting long-running async SQS polling).

Changes:

  • Apply request_timeout to the curl handle via pycurl.TIMEOUT.
  • Apply connect_timeout to the curl handle via pycurl.CONNECTTIMEOUT.
  • Document rationale in-code (network partitions / proxy termination leading to hangs).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread kombu/asynchronous/http/curl.py Outdated
auvipy and others added 2 commits March 15, 2026 18:50
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont we need any unit tests for the changes proposed?

@auvipy auvipy added this to the 5.7.0 milestone Mar 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants