Skip to content

CURL-INTEGRATION.md

Latest commit

 

History

History
153 lines (110 loc) · 6.67 KB

CURL-INTEGRATION.md

File metadata and controls

153 lines (110 loc) · 6.67 KB

cURL Async Integration — Known Issues & Architecture

This document covers important technical details about the async cURL integration, including known libcurl bugs and the workarounds applied.

Overview

The async cURL integration uses libcurl's multi_socket API combined with the PAUSE/unpause pattern for non-blocking I/O:

  • File uploads (CURLFile): curl_mime_data_cb() with a read callback that returns CURL_READFUNC_PAUSE while async file I/O is in progress.
  • File downloads (CURLOPT_FILE): write callback returns CURL_WRITEFUNC_PAUSE while async file write is in progress.
  • User callbacks (CURLOPT_WRITEFUNCTION): write callback pauses, spawns a high-priority coroutine to run the PHP callback, then unpauses.

After async I/O completes, the transfer is unpaused via curl_easy_pause(CURLPAUSE_CONT) and driven forward with curl_multi_socket_action(CURL_SOCKET_TIMEOUT).

Minimum libcurl Version

Recommended: libcurl >= 8.11.1 for fully async file upload support.

On older versions, file uploads (CURLFile) fall back to synchronous read() inside the read callback. This is safe for local files but blocks the event loop briefly during each read. Downloads and user write callbacks work correctly on all versions.

The PAUSE/unpause Bug (libcurl < 8.11.1)

Symptom

Intermittent timeout on file uploads (~20% failure rate):

Operation timed out after 5000 milliseconds with 0 bytes received

Root Cause

Multiple bugs in libcurl's PAUSE/unpause mechanism prevent the transfer from being driven after curl_easy_pause(CURLPAUSE_CONT):

  1. timer_lastcall / last_expire_ts optimization — Curl_update_timer() skips the timer callback when the new expire timestamp matches the cached one. Fixed in curl#15627 (8.11.1), but the fix was removed during intermediate refactors (present in 8.5.0, absent in 8.6–8.10, re-added in 8.11.1).

  2. tempcount guard on cselect_bits — In curl_easy_pause(), data->conn->cselect_bits is only set when data->state.tempcount == 0. If any response data arrived while the transfer was paused, tempcount > 0 and cselect_bits is never set. Without cselect_bits, the transfer is not processed even when timeouts fire correctly. Fixed in 8.11.1+ where cselect_bits was replaced with data->state.select_bits (always set, no tempcount guard).

  3. CURLINFO_ACTIVESOCKET unreliable during transfer — Returns CURL_SOCKET_BAD (-1) in the multi_socket API because lastconnect_id is not set until the transfer completes. Cannot be used to drive the socket directly.

Workarounds Tested (all insufficient for < 8.11.1)

Approach Result
curl_multi_socket_action(CURL_SOCKET_TIMEOUT) after unpause ~80% pass
Manual curl_timer_cb(multi, 0, NULL) to force 0ms timer ~94% pass
CURLINFO_ACTIVESOCKET + CURL_CSELECT_IN|OUT ~92% pass (socket sometimes BAD)
Track socket via curl_socket_cb + direct socket action ~82–92% pass (socket removed from sockhash during pause)
curl_multi_perform() ~74% pass (must not mix with multi_socket API)

Solution Applied

For libcurl < 8.11.1, the file upload read callback (curl_async_read_cb) uses synchronous read() instead of the async PAUSE/unpause pattern. This completely avoids the bug — no PAUSE means no broken unpause.

#if LIBCURL_VERSION_NUM < 0x080B01
    // Synchronous read — safe for local files
    const ssize_t n = read(fd, buffer, requested);
#else
    // Async PAUSE/unpause pattern
    return CURL_READFUNC_PAUSE;
#endif

Compile-time check via LIBCURL_VERSION_NUM. Zero runtime overhead. With libcurl >= 8.11.1, the async path is used and works 100% reliably.

Timer Callback Regression (libcurl 8.10.x)

Symptom

curl_exec() hangs indefinitely when connecting to unreachable hosts (e.g. 192.0.2.1), even with CURLOPT_TIMEOUT and CURLOPT_CONNECTTIMEOUT set. The timeout never fires in the multi_socket API.

Root Cause

A regression introduced in libcurl 8.10.0 causes CURLMOPT_TIMERFUNCTION to not be called again after curl_multi_socket_action(CURL_SOCKET_TIMEOUT). The sequence:

  1. curl registers a connect socket and requests a timeout timer (e.g. 2000ms)
  2. Timer fires → curl_multi_socket_action(CURL_SOCKET_TIMEOUT) is called
  3. curl returns running_handles=1 — does not complete the transfer
  4. curl does not call CURLMOPT_TIMERFUNCTION to request a new timer
  5. The socket never becomes writable (host unreachable) → deadlock

In working versions (e.g. 8.5.0), step 4 correctly re-arms the timer with a short interval (~35ms), which fires again and completes the transfer with CURLE_OPERATION_TIMEDOUT.

Affected Versions

  • Broken: libcurl 8.10.0 – 8.11.0
  • Working: libcurl ≤ 8.9.1, libcurl ≥ 8.11.1
  • Related: curl#15154 — "curl_multi still breaks in 8.10.1 with libev"
  • Related: curl#15639 — Regression in multi-curl async calls (8.9.1 → 8.10.0/8.11.0)
  • Fix: curl#15627 — merged into curl 8.11.1

Recommendation

Avoid libcurl 8.10.x for any multi_socket-based integration. Use libcurl 8.5.0 (as in CI) or >= 8.11.1.

References

Files

  • ext/curl/curl_async.c — Async cURL implementation (read/write callbacks, unpause logic)
  • ext/curl/curl_async.h — Struct definitions and public API
  • ext/curl/interface.c — build_mime_structure_from_hash() registers the async read callback
  • ext/curl/curl_private.h — mime_data_cb_arg_t struct with async state