Skip to content

ThreadPool constructor might fail when pthread_create fails leaving threads unjoined #2444

@carlopi

Description

@carlopi

We are using httplib in duckdb, and we are super happy with it.

In an experimental branch, I try to use httplib's ThreadPool, but I found out that when (due to resource exhaustion in the system I was running the tests) pthread_create fails, the constructor for ThreadPool might leave threads in inconsistent state, and, in the context of my program, that lead to a std::terminate.

I am not completely sure this makes sense when outside duckdb / my specific branch, but I though of raising.

I am considering patching duckdb's httplib (would need to discuss with maintainers there) like:

@@ -904,9 +904,26 @@ class ThreadPool final : public TaskQueue {
 public:
   explicit ThreadPool(size_t n, size_t mqr = 0)
       : shutdown_(false), max_queued_requests_(mqr) {
-    while (n) {
-      threads_.emplace_back(worker(*this));
-      n--;
+    // Exception-safe construction: if std::thread creation fails partway
+    // (e.g. EAGAIN on pthread_create under thread-resource pressure), the
+    // partially-built threads_ vector would otherwise destruct with joinable
+    // std::thread objects, calling std::terminate(). Instead, signal shutdown
+    // to the workers we already spawned, join them, and rethrow.
+    try {
+      while (n) {
+        threads_.emplace_back(worker(*this));
+        n--;
+      }
+    } catch (...) {
+      {
+        duckdb::unique_lock<std::mutex> lock(mutex_);
+        shutdown_ = true;
+      }
+      cond_.notify_all();
+      for (auto &t : threads_) {
+        if (t.joinable()) { t.join(); }
+      }
+      throw;
     }
   }
 

see the diff at https://github.com/duckdb/duckdb/pull/22516/changes, now looking for guidance, it's this a proper httplib issue? Should I file a similar PR to httplib instead, would that make sense?

Thanks for the great project, and for any guidance you might offer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions