Skip to content

Commit 657a1bc

Browse files
committed
fix(PrimaryClient): close stream before joining reconnect thread; interruptible producer backoff
Extends the RTDEClient teardown fix to the PrimaryClient pipeline, which is the path used by UrDriver. ~PrimaryClient() could block indefinitely (up to the unbounded reconnect timeout) when the robot dropped the primary connection at teardown time. Root cause: on connection loss TCPSocket::read() leaves the socket in SocketState::Disconnected, so URProducer::tryGetImpl() enters its reconnect loop (sleep backoff + stream_.connect() with unlimited retries). ~PrimaryClient() and PrimaryClient::stop() joined the producer thread (pipeline_->stop()) without first closing the stream, so the join blocked for the full reconnect duration. Two related fixes: 1. ~PrimaryClient() and PrimaryClient::stop() now call stream_.close() BEFORE pipeline_->stop(). stream_.close() sets SocketState::Closed, waking a producer stuck in its reconnect path so the join returns promptly. Previously stop() closed the stream after the join (too late) and the destructor never closed it. 2. URProducer::tryGetImpl()'s reconnect backoff slept sleep_for(timeout_) (growing up to 120 s) with no cancellation point. It now sleeps in 100 ms slices and bails out as soon as running_ becomes false or the stream is closed, mirroring the TCPSocket::setup() interruptible-sleep fix. test: add PrimaryClientReconnectTest.destructor_not_blocked_by_stuck_reconnect_thread (test_primary_client_reconnect.cpp). The PrimaryClient counterpart of RTDEClientTest.destructor_not_blocked_by_stuck_reconnect_thread, using the in-process FakePrimaryServer so it runs without a robot (unlike the INTEGRATION_TESTS-gated primary_client_test_headless). It starts a client against the fake server, drops the server to drive the producer into its reconnect loop, then asserts ~PrimaryClient() completes in < 2 s. Verified to hang without the fix and pass in ~1.5 s with it. Co-authored-by: Rune Søe-Knudsen <urrsk@users.noreply.github.com>
1 parent 209b5f5 commit 657a1bc

4 files changed

Lines changed: 141 additions & 2 deletions

File tree

include/ur_client_library/comm/producer.h

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,21 @@ class URProducer : public IProducer<T>
8585
}
8686

8787
URCL_LOG_WARN("Failed to read from stream, reconnecting in %ld seconds...", timeout_.count());
88-
std::this_thread::sleep_for(timeout_);
88+
// Sleep in small slices so the producer can be stopped (running_ == false)
89+
// or woken by a stream close (e.g. from a destructor calling stop()) within
90+
// ~100 ms, instead of blocking for the full (exponentially growing) timeout.
91+
// Without this, a thread joining the pipeline at teardown could block for up
92+
// to 120 s while this thread sleeps here.
93+
const auto sleep_slice = std::chrono::milliseconds(100);
94+
const auto sleep_total = std::chrono::duration_cast<std::chrono::milliseconds>(timeout_);
95+
for (auto slept = std::chrono::milliseconds(0); slept < sleep_total && running_ && !stream_.closed();
96+
slept += sleep_slice)
97+
{
98+
std::this_thread::sleep_for(sleep_slice);
99+
}
100+
101+
if (!running_ || stream_.closed())
102+
return false;
89103

90104
if (stream_.connect())
91105
continue;

src/primary/primary_client.cpp

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,12 @@ PrimaryClient::PrimaryClient(const std::string& robot_ip, [[maybe_unused]] comm:
6767
PrimaryClient::~PrimaryClient()
6868
{
6969
URCL_LOG_INFO("Stopping primary client pipeline");
70+
// Close the stream BEFORE stopping (joining) the pipeline. The pipeline's
71+
// producer thread may be sleeping inside its reconnect backoff or blocked in
72+
// TCPSocket::setup(); closing the stream sets SocketState::Closed, which wakes
73+
// both paths so pipeline_->stop()'s join returns promptly instead of blocking
74+
// until the (potentially unbounded) reconnect timeout expires.
75+
stream_.close();
7076
pipeline_->stop();
7177
}
7278

@@ -79,8 +85,10 @@ void PrimaryClient::start(const size_t max_num_tries, const std::chrono::millise
7985

8086
void PrimaryClient::stop()
8187
{
82-
pipeline_->stop();
88+
// Close the stream before joining the pipeline so a producer thread stuck in
89+
// its reconnect path is woken and the join returns promptly (see ~PrimaryClient).
8390
stream_.close();
91+
pipeline_->stop();
8492
}
8593

8694
void PrimaryClient::addPrimaryConsumer(std::shared_ptr<comm::IConsumer<PrimaryPackage>> primary_consumer)

tests/CMakeLists.txt

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,14 @@ target_link_libraries(fake_primary_server_tests PRIVATE ur_client_library::urcl
152152
gtest_add_tests(TARGET fake_primary_server_tests
153153
)
154154

155+
# Robot-free regression test for ~PrimaryClient() not blocking on a stuck
156+
# reconnect thread. Uses the in-process FakePrimaryServer so it runs without a
157+
# robot (unlike the INTEGRATION_TESTS-gated primary_client_test_headless).
158+
add_executable(primary_client_reconnect_tests test_primary_client_reconnect.cpp fake_primary_server.cpp)
159+
target_link_libraries(primary_client_reconnect_tests PRIVATE ur_client_library::urcl GTest::gtest_main)
160+
gtest_add_tests(TARGET primary_client_reconnect_tests
161+
)
162+
155163

156164

157165
add_executable(rtde_data_package_tests test_rtde_data_package.cpp)
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
// -- BEGIN LICENSE BLOCK ----------------------------------------------
2+
// Copyright 2026 Universal Robots A/S
3+
//
4+
// Redistribution and use in source and binary forms, with or without
5+
// modification, are permitted provided that the following conditions are met:
6+
//
7+
// * Redistributions of source code must retain the above copyright
8+
// notice, this list of conditions and the following disclaimer.
9+
//
10+
// * Redistributions in binary form must reproduce the above copyright
11+
// notice, this list of conditions and the following disclaimer in the
12+
// documentation and/or other materials provided with the distribution.
13+
//
14+
// * Neither the name of the {copyright_holder} nor the names of its
15+
// contributors may be used to endorse or promote products derived from
16+
// this software without specific prior written permission.
17+
//
18+
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
19+
// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
20+
// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
21+
// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
22+
// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
23+
// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
24+
// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
25+
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
26+
// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
27+
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
28+
// POSSIBILITY OF SUCH DAMAGE.
29+
// -- END LICENSE BLOCK ------------------------------------------------
30+
31+
#include <gtest/gtest.h>
32+
33+
#include <chrono>
34+
#include <memory>
35+
#include <thread>
36+
37+
#include <ur_client_library/primary/primary_client.h>
38+
39+
#include "fake_primary_server.h"
40+
41+
using namespace urcl;
42+
43+
// Regression test for ~PrimaryClient() blocking indefinitely when the pipeline's
44+
// producer thread is stuck in its reconnect loop at teardown time.
45+
//
46+
// This is the PrimaryClient counterpart of
47+
// RTDEClientTest.destructor_not_blocked_by_stuck_reconnect_thread (test_rtde_client.cpp).
48+
//
49+
// Root cause: when the robot drops the primary connection, TCPSocket::read()
50+
// returns false and leaves the socket in SocketState::Disconnected. URProducer's
51+
// tryGetImpl() then enters its reconnect path: it sleeps an (exponentially
52+
// growing) backoff and calls stream_.connect(), which retries with no upper
53+
// bound (max_num_tries == 0), sleeping reconnection_time between attempts. If
54+
// ~PrimaryClient() simply called pipeline_->stop() (which joins the producer
55+
// thread) without first closing the stream, the join would block for the full
56+
// reconnect duration — effectively forever for an unreachable robot.
57+
//
58+
// Fix (two parts):
59+
// 1. ~PrimaryClient()/PrimaryClient::stop() close the stream BEFORE joining the
60+
// pipeline. stream_.close() sets SocketState::Closed.
61+
// 2. URProducer's reconnect backoff sleeps in 100 ms slices and bails out as
62+
// soon as the stream is closed (or the producer is stopped), and
63+
// TCPSocket::setup()'s between-attempt sleep is likewise interruptible by
64+
// SocketState::Closed.
65+
// Together these wake the producer within ~100 ms of the destructor closing the
66+
// stream, so the join — and therefore the destructor — returns promptly.
67+
//
68+
// Unlike test_primary_client.cpp's robot-dependent fixtures, this test uses the
69+
// in-process FakePrimaryServer, so it runs in the normal (non-INTEGRATION_TESTS)
70+
// build and needs no robot.
71+
TEST(PrimaryClientReconnectTest, destructor_not_blocked_by_stuck_reconnect_thread)
72+
{
73+
comm::INotifier notifier;
74+
75+
auto server = std::make_unique<FakePrimaryServer>(primary_interface::UR_PRIMARY_PORT);
76+
auto client = std::make_unique<primary_interface::PrimaryClient>("127.0.0.1", notifier);
77+
78+
// Unlimited reconnect attempts with a large reconnection time: if the fix is
79+
// absent, the producer's reconnect path keeps the destructor blocked.
80+
const std::chrono::milliseconds large_reconnect_timeout(5000);
81+
ASSERT_NO_THROW(client->start(/*max_num_tries=*/0, large_reconnect_timeout));
82+
ASSERT_TRUE(server->waitForClient()) << "PrimaryClient never connected to the fake server";
83+
84+
// Drop the server. The producer's read() fails, the socket transitions to
85+
// SocketState::Disconnected, and the producer enters its reconnect loop.
86+
server.reset();
87+
88+
// Give the producer time to detect the drop and reach its reconnect sleep
89+
// (initial backoff is 1 s, after which it sleeps inside TCPSocket::setup()).
90+
std::this_thread::sleep_for(std::chrono::milliseconds(1500));
91+
92+
// The destructor must return quickly: it closes the stream
93+
// (SocketState::Closed), waking the producer, so the pipeline join completes in
94+
// well under 2 s. Without the fix this blocks for at least the reconnect
95+
// timeout (and indefinitely with unlimited retries against a dead port).
96+
const auto t0 = std::chrono::steady_clock::now();
97+
client.reset();
98+
const auto elapsed = std::chrono::steady_clock::now() - t0;
99+
100+
EXPECT_LT(elapsed, std::chrono::seconds(2))
101+
<< "~PrimaryClient() blocked for " << std::chrono::duration_cast<std::chrono::milliseconds>(elapsed).count()
102+
<< " ms — the producer reconnect thread was not woken by the stream close";
103+
}
104+
105+
int main(int argc, char* argv[])
106+
{
107+
::testing::InitGoogleTest(&argc, argv);
108+
return RUN_ALL_TESTS();
109+
}

0 commit comments

Comments
 (0)