We are using httplib in duckdb, and we are super happy with it.
In an experimental branch, I try to use httplib's ThreadPool, but I found out that when (due to resource exhaustion in the system I was running the tests) pthread_create fails, the constructor for ThreadPool might leave threads in inconsistent state, and, in the context of my program, that lead to a std::terminate.
I am not completely sure this makes sense when outside duckdb / my specific branch, but I though of raising.
I am considering patching duckdb's httplib (would need to discuss with maintainers there) like:
@@ -904,9 +904,26 @@ class ThreadPool final : public TaskQueue {
public:
explicit ThreadPool(size_t n, size_t mqr = 0)
: shutdown_(false), max_queued_requests_(mqr) {
- while (n) {
- threads_.emplace_back(worker(*this));
- n--;
+ // Exception-safe construction: if std::thread creation fails partway
+ // (e.g. EAGAIN on pthread_create under thread-resource pressure), the
+ // partially-built threads_ vector would otherwise destruct with joinable
+ // std::thread objects, calling std::terminate(). Instead, signal shutdown
+ // to the workers we already spawned, join them, and rethrow.
+ try {
+ while (n) {
+ threads_.emplace_back(worker(*this));
+ n--;
+ }
+ } catch (...) {
+ {
+ duckdb::unique_lock<std::mutex> lock(mutex_);
+ shutdown_ = true;
+ }
+ cond_.notify_all();
+ for (auto &t : threads_) {
+ if (t.joinable()) { t.join(); }
+ }
+ throw;
}
}
see the diff at https://github.com/duckdb/duckdb/pull/22516/changes, now looking for guidance, it's this a proper httplib issue? Should I file a similar PR to httplib instead, would that make sense?
Thanks for the great project, and for any guidance you might offer.
We are using httplib in duckdb, and we are super happy with it.
In an experimental branch, I try to use
httplib'sThreadPool, but I found out that when (due to resource exhaustion in the system I was running the tests) pthread_create fails, the constructor forThreadPoolmight leave threads in inconsistent state, and, in the context of my program, that lead to astd::terminate.I am not completely sure this makes sense when outside duckdb / my specific branch, but I though of raising.
I am considering patching duckdb's httplib (would need to discuss with maintainers there) like:
see the diff at https://github.com/duckdb/duckdb/pull/22516/changes, now looking for guidance, it's this a proper
httplibissue? Should I file a similar PR to httplib instead, would that make sense?Thanks for the great project, and for any guidance you might offer.