[DISCUSS] Selecting a C++ HTTP Client Library for REST

[HeartLinked]

I'm proposing we select an HTTP client library for rest catalog. I've reviewed libcurl, cpr, and cpp-httplib.

I suggest we rule out libcurl for direct use. Its C-style API is too complex and error-prone for a modern C++23 project.This leaves us with two primary contenders that offer modern C++ wrappers: cpr and cpp-httplib.The main advantage of cpp-httplib is its simplicity. It's a header-only library with virtually zero dependencies (unless HTTPS is needed), which makes integration extremely straightforward.

However, after a deeper comparison, I believe cpr is the superior choice for our project for its modern and usable API, rich features. Because cpr is a wrapper around libcurl, it inherits its robustness and extensive feature set. This includes superior performance characteristics, fine-grained control over things like timeouts and authentication, and asynchronous request capabilities.

I've conducted a POC with a demo using integrated cpr here: #230 . WDYT?

[wgtmac]

@dongxiao1198 @gty404 @lidavidm @mapleFU @yingcai-cy @zhjwpku WDYT?

[acking-you]

Very interesting proposal. It feels like cpr is quite good, but I'm curious about how package management is currently handled in iceberg-cpp. Can it adopt something as convenient as vcpkg, like arrow-cpp does?

[dongxiao1198]

Libcurl is a low-level common lib,has a lot of settings, but it is stable & small & no-extra dependency.
CPR is much more modern and easy to use, but itself is a 3rd lib, we need to keep upgrading it in the future(libcurl is much more stable)
Poco is also a good choice lots of rest api client use it.(but it is too heavy for us?)

Anyway, since we just play as a simple rest client, I prefer to use a small & lightweight lib such as libcurl or cpr.

PS: Can we make it pluggable？

[lidavidm]

Is there any benefit to pluggability here (is there any real alternative? cronet? OS APIs?)

[acking-you]

CPR is much more modern and easy to use, but itself is a 3rd lib, we need to keep upgrading it in the future(libcurl is much more stable)

If package management is convenient enough, I don't think continuous upgrades are a bad thing.

[zhjwpku]

Based on your description and a brief search, I'm +1 for adopting cpr, since its API is easier to use compared to libcurl's C-only interface, at this early stage, I think we should rapidly get things working, we can always revisit and refactor later if needed.

[HeartLinked]

The POC for integrating cpr #236 has now passed all CI checks, successfully completing the integration work.

[wgtmac]

Closing this as #236 is merged