Skip to content

Commit 2f9d3e1

Browse files
committed
Fix event-loop freeze: try_to_lock in IO-thread tracker accesses
Root cause of recurring SIGABRT cycle on contabo (2026-04-12, -16, -19, -21, -25): three IO-thread paths used blocking std::shared_lock on m_tracker_mutex when the architectural rule (node.hpp:67) requires try_to_lock everywhere on the IO thread. When the compute thread held the exclusive write lock for a long think+clean cycle on a wedged chain (~30+s), the next incoming SHAREREQ would block here, the ext_watchdog would fire after 30s of io_context unresponsive, and systemd would restart. Confirmed via core dump from contabo's 2026-04-25 20:31 crash: Thread 6 (main IO) was caught in pthread_rwlock_rdlock_full64 inside ltc::NodeImpl::handle_get_share at node.cpp:581. The compute thread had already released by the time of SIGABRT but Thread 6 was still stuck mid-block. Three fixes — all the same shape (try_to_lock + log + return on busy): - handle_get_share — empty reply OK, peer's downloader picks another random peer next iteration (p2pool node.py:120) - send_shares — defer to next broadcast cycle - broadcast_share — defer; share is still in chain, next think / SHAREREQ will pick it up This addresses the freeze at the source: think+clean can take as long as it likes on a wedged chain, the IO thread will keep flowing.
1 parent a53613d commit 2f9d3e1

1 file changed

Lines changed: 52 additions & 9 deletions

File tree

src/impl/ltc/node.cpp

Lines changed: 52 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -576,9 +576,28 @@ void NodeImpl::processing_shares_phase2(HandleSharesData& data, NetService addr)
576576

577577
std::vector<ltc::ShareType> NodeImpl::handle_get_share(std::vector<uint256> hashes, uint64_t parents, std::vector<uint256> stops, NetService peer_addr)
578578
{
579-
// Shared lock: waits for think() to release exclusive lock (200-400ms max).
580-
// Must respond to share requests — returning empty causes peer disconnect.
581-
std::shared_lock lock(m_tracker_mutex);
579+
// try_to_lock per the architectural rule (node.hpp:67) — IO thread MUST
580+
// never block on m_tracker_mutex. A blocking shared_lock here was the
581+
// root cause of the periodic event-loop freeze + SIGABRT cycle on
582+
// contabo (2026-04-12, -16, -19, -21, -25): when the compute thread
583+
// held the exclusive lock for a long think+clean cycle on a wedged
584+
// chain (~30+s), an incoming SHAREREQ on the IO thread would block
585+
// here, the watchdog would fire after 30s of io_context unresponsive,
586+
// and systemd would restart.
587+
//
588+
// Empty reply does NOT cause peer disconnect — p2pool's downloader
589+
// (node.py:120) picks a random peer per request and retries; an empty
590+
// hit just shifts to a different peer next iteration.
591+
std::shared_lock<std::shared_mutex> lock(m_tracker_mutex, std::try_to_lock);
592+
if (!lock.owns_lock())
593+
{
594+
static int defer_log = 0;
595+
if (defer_log++ % 50 == 0)
596+
LOG_INFO << "[handle_get_share] tracker busy — returning empty to "
597+
<< peer_addr.to_string()
598+
<< " (peer will retry against another peer)";
599+
return {};
600+
}
582601

583602
parents = std::min(parents, (uint64_t)1000/hashes.size());
584603
std::vector<ltc::ShareType> shares;
@@ -614,9 +633,19 @@ std::vector<ltc::ShareType> NodeImpl::handle_get_share(std::vector<uint256> hash
614633

615634
void NodeImpl::send_shares(peer_ptr peer, const std::vector<uint256>& share_hashes)
616635
{
617-
// Shared lock: waits for think() to release (200-400ms max).
618-
// Must send shares — skipping causes peer to see us as empty.
619-
std::shared_lock lock(m_tracker_mutex);
636+
// try_to_lock per the architectural rule (node.hpp:67) — see freeze
637+
// analysis in handle_get_share above. If we can't acquire NOW, skip
638+
// this batch. The shares are still in our chain; the next broadcast
639+
// cycle (or the next think() result) picks them up.
640+
std::shared_lock<std::shared_mutex> lock(m_tracker_mutex, std::try_to_lock);
641+
if (!lock.owns_lock())
642+
{
643+
static int defer_log = 0;
644+
if (defer_log++ % 50 == 0)
645+
LOG_INFO << "[send_shares] tracker busy — skipping send to "
646+
<< peer->addr().to_string() << " (will retry next cycle)";
647+
return;
648+
}
620649

621650
// Collect shares that exist in our chain (skip rejected)
622651
std::vector<ShareType> shares;
@@ -714,9 +743,23 @@ void NodeImpl::send_shares(peer_ptr peer, const std::vector<uint256>& share_hash
714743

715744
void NodeImpl::broadcast_share(const uint256& share_hash)
716745
{
717-
// Shared lock: waits for think() to release (200-400ms max).
718-
// Must broadcast — skipping causes share propagation stalls.
719-
std::shared_lock lock(m_tracker_mutex);
746+
// try_to_lock per the architectural rule (node.hpp:67) — see freeze
747+
// analysis in handle_get_share above. If think+clean holds the
748+
// exclusive lock right now, defer this broadcast: the share is still
749+
// in our chain; the next local share creation, the next think cycle,
750+
// or the next peer-driven SHAREREQ will pick it up. Blocking here
751+
// was a contributing freeze trigger (called from local-share creation
752+
// and from the share-add hot path).
753+
std::shared_lock<std::shared_mutex> lock(m_tracker_mutex, std::try_to_lock);
754+
if (!lock.owns_lock())
755+
{
756+
static int defer_log = 0;
757+
if (defer_log++ % 50 == 0)
758+
LOG_INFO << "[broadcast_share] tracker busy — deferring broadcast of "
759+
<< share_hash.GetHex().substr(0, 16)
760+
<< " (next cycle will pick it up)";
761+
return;
762+
}
720763

721764
// Walk the chain back from share_hash, collecting un-broadcast shares
722765
std::vector<uint256> to_send;

0 commit comments

Comments
 (0)