dash: decouple blocking coin-RPC + heavy work off the stratum io_context (io-thread-decouple)#781
Merged
Merged
Conversation
added 4 commits
July 21, 2026 01:01
The DASH hotel node stalled stratum under load (~66 conns from ~23 rigs x 2 pools): main_dash ran StratumServer, coin-P2P, the blocking NodeRPC and the 3 s tip-poll's getbestblockhash all on ONE io_context thread, so the io thread saturated and marginal sessions starved. Port the proven LTC decouple (thread_pool(1) + post-back-to-ioc) into DASH. F1 (zombie-session leak, the acute N-collapse): a NAT-dropped rig's TCP session is never FIN/RST'd, so socket_.is_open() never falsifies and the session is never reaped -- every failed retry minted an immortal subscribed session drawing full per-notify job builds. Add OS TCP keepalive, an authorize-handshake deadline, an idle reaper and a write-queue cap to core StratumServer, all gated by new StratumConfig knobs that DEFAULT OFF (LTC/BTC/DGB byte-unchanged); DASH opts in. F2 (structural decouple): a dedicated single-thread rpc_pool runs the fallback arm's blocking getbestblockhash and posts the tip-change follow-up (invalidate + bump + notify) back onto ioc; NodeRPC::Send is serialized with a mutex (io thread getwork vs pool thread getbestblockhash share one beast client) and gets a kernel SO_SND/RCVTIMEO deadline so a wedged dashd can't hang the pool thread. cached_work() serves the cached template immediately and hands the blocking re-source to the rpc_pool as a single-flight background job, so a per-share generation bump no longer blocks the io thread on a full GBT. The rpc_pool is stopped+joined before its referents unwind at shutdown. Consensus-neutral (threading/transport only; the select_work/payment logic is moved byte-identical into resource_template_now). KAT added: IoThreadDecoupleServesCachedAndRefreshesOffThread.
… real Send() deadline, keepalive-aware reaper
CRITICAL-1: commit the Boost-1.90 idle_reap_timer_.cancel() fix (was left
uncommitted, so every CI lane was red at stratum_server.cpp:138 and no KAT ran
in CI). Swept the tree: all timer .cancel() calls are the no-arg overload.
CRITICAL-2: port LTC's m_rpc_mutex verbatim (impl/ltc/coin/rpc.{hpp,cpp}) into
dash::coin::NodeRPC -- the prior commit's claim was real but the edit never
landed. Send() now opens with std::lock_guard(m_rpc_mutex); serializing the
shared m_http_request/beast::tcp_stream across the io thread (getwork /
submit_block_hex) and the rpc_pool thread (getbestblockhash / GBT refresh). This
closes the won-block-path UAF / cross-wired-submitblock race (tip change on a won
block = pool thread very likely mid-RPC).
HIGH: implement a REAL Send() deadline. beast's synchronous read_some calls
socket.read_some() directly and ignores tcp_stream::expires_after (async-only),
so apply_socket_timeouts() forces the socket back to blocking mode (async_connect
leaves asio non_blocking set, under which SO_*TIMEO is a no-op) and sets kernel
SO_RCV/SNDTIMEO (12 s). A wedged dashd now returns an error instead of hanging
the pool thread forever -> template_refresh_inflight_ clears, the tip-poll
re-arms, and rpc_pool->join() at shutdown terminates.
MED: idle reaper 600 -> 1800 s AND keepalive-aware -- when OS keepalive is on it
is the liveness authority, so an authorized rig whose socket is still keepalive-
validated is never reaped for idleness (protects high fixed-diff ADDR+N rigs
from being cycled). Idle-time reaping is the fallback only where keepalive is
unavailable / for never-authorized sessions.
Consensus-neutral (threading/transport only). KATs: 34/34 green.
The Windows x86_64 lane (MSVC core.vcxproj) failed C2664: Winsock's setsockopt takes const char* for the value arg, so the int* value args in StratumSession::apply_socket_keepalive()'s TCP_KEEPIDLE/INTVL/CNT calls did not convert. The per-probe keepalive tuning is Linux-specific anyway (Windows uses WSAIoctl(SIO_KEEPALIVE_VALS), macOS uses TCP_KEEPALIVE), so guard that block with #ifdef __linux__ exactly like the existing sample_tcp_rtt_ms(); the portable asio keep_alive(true) still arms keepalive with OS defaults off-Linux. The DASH deploy target is Linux, where the 60/10/3 (~90 s detect) tuning applies. Likewise guard NodeRPC::apply_socket_timeouts()'s SO_RCV/SNDTIMEO (needs <sys/time.h>/struct timeval, absent on Windows) with #ifndef _WIN32 so a Windows DASH build is a clean no-op; Linux and macOS keep the real Send() deadline. Transport/liveness only; Linux behaviour unchanged. KATs 34/34 green.
…(deadline-desync guard)
The SO_RCVTIMEO deadline added for the io-thread decouple introduced a new
failure class the pre-PR blocking code could not hit: on the retry loop's FINAL
attempt, a write-fail or read-fail did 'return {}' while leaving m_stream OPEN
with a request already written into dashd but its response unread. Because
jsonrpccxx reuses the constant id 'curltest' and never validates response ids,
the NEXT Send() on that reused connection would read the PREVIOUS call's late
response -> permanent off-by-one desync. On the hotel that means getblocktemplate
parses a getbestblockhash string -> zeroed work -> a stuck 'honest set-gap' with
the node serving NO work until the connection churns/restart. Trigger: a >=~24s
dashd stall spanning both 12s SO_RCVTIMEO attempts with zero response bytes.
Fix: add close_stream() (the sync_reconnect()/destructor teardown idiom:
socket shutdown_both + close + m_connected=false) and call it on both final-
attempt exit paths before 'return {}', so the next Send() write-fails into a
clean sync_reconnect() instead of desyncing. attempt-0 already tears down via
sync_reconnect(); only the final attempt leaked. Held under m_rpc_mutex.
Transport only; consensus-neutral. KATs 34/34 green.
frstrtr
marked this pull request as ready for review
July 21, 2026 00:28
This was referenced Jul 21, 2026
Closed
frstrtr
added a commit
that referenced
this pull request
Jul 21, 2026
…socket deadline (#744/#787) Arming ARM B (m_rpc) made four previously-dead defects live on the reward path. Review (Fable) findings, all fixed: B1 (blocker) — NodeRPC::check() probed the LITECOIN genesis hash (copied from the LTC impl, a #759-class cross-coin drift). On BTC mainnet the is_main_chain && !has_block gate always failed -> permanent 15s reconnect loop, ARM B degraded on the one net where a lost block is real money. New btc/coin/genesis.hpp + btc_genesis_hash(IS_TESTNET); check() now probes the per-net BTC genesis (mirrors the DGB reference). genesis_check_test.cpp pins it (regression guard vs the LTC hash). B2 (blocker) — unguarded std::stoi in rpc_conf.hpp aborted the node at startup on a malformed rpcport (e.g. rpcport=abc) in the DEFAULT ~/.bitcoin/bitcoin.conf, crashing operators who never opted in. New conf_detail::parse_port (digits-only, range-checked, never throws) -> junk degrades to UNARMED. 3 new KATs. M1 (major) — a daemon unreachable at submit raised JsonRpcException out of submit_block_hex; broadcast_block_for_connect had no throw guards (unlike the core SSOT), so the exception destroyed ARM A's already-succeeded relay -> false 'reached NEITHER -- lost subsidy' alarm for a block that WAS relayed. Both legs now guarded. 3 new throwing-leg KATs. M2 (major) — no socket deadline on the beast tcp_stream shared with stratum on one single-threaded ioc; a hung bitcoind froze the whole node mid-won-block. Added apply_socket_timeouts() (SO_SND/RCVTIMEO, mirrors DASH #781), armed after each (re)connect. Also softened the arm_submit_rpc comment (check() does issue read-only probes). Broadcast/RPC-transport path only -- no PoW, share, coinbase-commitment, or PPLNS math touched (consensus-neutral). BTC-fenced. Builds clean (c2pool-btc + btc_share_test); 20 KATs pass.
frstrtr
added a commit
that referenced
this pull request
Jul 21, 2026
…socket deadline (#744/#787) Arming ARM B (m_rpc) made four previously-dead defects live on the reward path. Review (review) findings, all fixed: B1 (blocker) — NodeRPC::check() probed the LITECOIN genesis hash (copied from the LTC impl, a #759-class cross-coin drift). On BTC mainnet the is_main_chain && !has_block gate always failed -> permanent 15s reconnect loop, ARM B degraded on the one net where a lost block is real money. New btc/coin/genesis.hpp + btc_genesis_hash(IS_TESTNET); check() now probes the per-net BTC genesis (mirrors the DGB reference). genesis_check_test.cpp pins it (regression guard vs the LTC hash). B2 (blocker) — unguarded std::stoi in rpc_conf.hpp aborted the node at startup on a malformed rpcport (e.g. rpcport=abc) in the DEFAULT ~/.bitcoin/bitcoin.conf, crashing operators who never opted in. New conf_detail::parse_port (digits-only, range-checked, never throws) -> junk degrades to UNARMED. 3 new KATs. M1 (major) — a daemon unreachable at submit raised JsonRpcException out of submit_block_hex; broadcast_block_for_connect had no throw guards (unlike the core SSOT), so the exception destroyed ARM A's already-succeeded relay -> false 'reached NEITHER -- lost subsidy' alarm for a block that WAS relayed. Both legs now guarded. 3 new throwing-leg KATs. M2 (major) — no socket deadline on the beast tcp_stream shared with stratum on one single-threaded ioc; a hung bitcoind froze the whole node mid-won-block. Added apply_socket_timeouts() (SO_SND/RCVTIMEO, mirrors DASH #781), armed after each (re)connect. Also softened the arm_submit_rpc comment (check() does issue read-only probes). Broadcast/RPC-transport path only -- no PoW, share, coinbase-commitment, or PPLNS math touched (consensus-neutral). BTC-fenced. Builds clean (c2pool-btc + btc_share_test); 20 KATs pass.
frstrtr
pushed a commit
that referenced
this pull request
Jul 21, 2026
…reward-safe --embedded-mainnet (default OFF) Rebased onto master (#781 io-decouple + #773/#777/#775/#782). Integrates the embedded DASH template arm into the decoupled tip-poll/rpc_pool architecture WHILE preserving both: #781's refresh_executor/resource_template_now split AND every v0.2.4 embedded guard. Default behaviour is UNCHANGED and reward-safe — with --embedded-mainnet OFF (the default) the template stays arm=dashd-fallback, the path the hotel runs, byte-for-byte. Wires the vendored SML/QuorumManager/CCbTx machinery into the live embedded template so the daemonless arm assembles a real DIP-0004 type-5 CCbTx, proven byte-identical to a real dashd getblocktemplate from the raw mnlistdiff wire (merkleRootMNList + merkleRootQuorums recomputed from wire; bestCL; creditPool accrual). The mainnet gate lifts behind the explicit --embedded-mainnet opt-in. Consensus guards (all fail-closed to the reward-safe dashd fallback): - merkleRootQuorums: hash ALL active commitments (no rotated per-index dedup) — byte-identical to dashd (109-quorum from-wire KAT). - H-1: a malformed quorum tail wipes + full-resyncs (never papered over by a later incremental; base-continuity tightened). - DKG mining-phase heights (is_dkg_commitment_window over every enabled llmqType) and superblock heights: refuse embedded. - bestCL freshness: committed ChainLock must be within one block of the tip. - creditPool: INDEPENDENT seed-height gate (the seed cbTx's own nHeight must equal the tip) — catches a seed one block behind that value/hash self-checks miss; pre-emit value re-check + serve-time cache re-validation for the build-vs-serve skew; GBT-xcheck backstop cross-checks the embedded creditPool vs dashd. - C-3: the embedded block excludes all DIP special txs (1-4,6,8,9) — their own-block state effects are not modelled in the committed CbTx (Phase-2). - pre-emit HARD GATE re-validates the built CbTx (both roots + all height-class + bestCL + creditPool checks) before serve; discards to fallback on any failure. Reward-safety posture: --embedded-mainnet is default OFF (unconfigured mainnet nodes are byte-unchanged dashd-fallback). The embedded-ON daemonless seed-lag (arm falls back rather than serving stale at some heights) is deferred as a separate goal; ON is reward-safe (worst case = fallback, never a bad block). Tests (dash embedded suite): node_coin_state 15, coin_state_maintainer 11, stratum_work_source 38, embedded_gbt 10, mnlistdiff_root_parity 4, byte_parity 4, quorum_root 11, mempool 19, node_reception_wire 21, header_chain 8. c2pool-dash builds+links against the #781 io-decoupled main_dash.
frstrtr
added a commit
that referenced
this pull request
Jul 21, 2026
ltc: Send() socket deadline + desync-safe teardown (parity with dash #781)
frstrtr
pushed a commit
that referenced
this pull request
Jul 21, 2026
…reward-safe --embedded-mainnet (default OFF) Rebased onto master (#781 io-decouple + #773/#777/#775/#782). Integrates the embedded DASH template arm into the decoupled tip-poll/rpc_pool architecture WHILE preserving both: #781's refresh_executor/resource_template_now split AND every v0.2.4 embedded guard. Default behaviour is UNCHANGED and reward-safe — with --embedded-mainnet OFF (the default) the template stays arm=dashd-fallback, the path the hotel runs, byte-for-byte. Wires the vendored SML/QuorumManager/CCbTx machinery into the live embedded template so the daemonless arm assembles a real DIP-0004 type-5 CCbTx, proven byte-identical to a real dashd getblocktemplate from the raw mnlistdiff wire (merkleRootMNList + merkleRootQuorums recomputed from wire; bestCL; creditPool accrual). The mainnet gate lifts behind the explicit --embedded-mainnet opt-in. Consensus guards (all fail-closed to the reward-safe dashd fallback): - merkleRootQuorums: hash ALL active commitments (no rotated per-index dedup) — byte-identical to dashd (109-quorum from-wire KAT). - H-1: a malformed quorum tail wipes + full-resyncs (never papered over by a later incremental; base-continuity tightened). - DKG mining-phase heights (is_dkg_commitment_window over every enabled llmqType) and superblock heights: refuse embedded. - bestCL freshness: committed ChainLock must be within one block of the tip. - creditPool: INDEPENDENT seed-height gate (the seed cbTx's own nHeight must equal the tip) — catches a seed one block behind that value/hash self-checks miss; pre-emit value re-check + serve-time cache re-validation for the build-vs-serve skew; GBT-xcheck backstop cross-checks the embedded creditPool vs dashd. - C-3: the embedded block excludes all DIP special txs (1-4,6,8,9) — their own-block state effects are not modelled in the committed CbTx (Phase-2). - pre-emit HARD GATE re-validates the built CbTx (both roots + all height-class + bestCL + creditPool checks) before serve; discards to fallback on any failure. Reward-safety posture: --embedded-mainnet is default OFF (unconfigured mainnet nodes are byte-unchanged dashd-fallback). The embedded-ON daemonless seed-lag (arm falls back rather than serving stale at some heights) is deferred as a separate goal; ON is reward-safe (worst case = fallback, never a bad block). Tests (dash embedded suite): node_coin_state 15, coin_state_maintainer 11, stratum_work_source 38, embedded_gbt 10, mnlistdiff_root_parity 4, byte_parity 4, quorum_root 11, mempool 19, node_reception_wire 21, header_chain 8. c2pool-dash builds+links against the #781 io-decoupled main_dash. CI-fold: build.yml "Build tests" allowlists test_dash_embedded_gbt but not the two byte-parity executables, so CTest registered them (gtest_add_tests AUTO) while CI never built them -> ctest "Not Run" exit 8 (Linux x86_64 red), and the from-wire merkleRoot + real-dashd CbTx round-trip proofs ran in NO CI job. Folded test_dash_embedded_cbtx_byte_parity.cpp + test_dash_mnlistdiff_root_parity.cpp INTO the test_dash_embedded_gbt executable (distinct TEST suites; DASH_FIXTURE_DIR moved to that target) and removed the standalone add_executable/gtest_add_tests registrations. test_dash_embedded_gbt 10 -> 18 cases (byte-parity 4 + from-wire 4), all green; no unbuilt-executable CTest registrations remain.
frstrtr
added a commit
that referenced
this pull request
Jul 22, 2026
…744) (#787) * btc: arm submitblock RPC backup leg (ARM B) of the dual-path broadcaster (#744) The embedded coin-net P2P relay (submit_block_for_connect -> submit_block_p2p_raw) is the always-primary daemonless broadcast arm and was already live-wired. The submitblock RPC backup (ARM B) existed in btc::coin::Node but m_rpc was never constructed in the run path (init_rpc was dead code that also forced getwork), so a won block could not reach a locally-run bitcoind -- the #744 gap. Add an opt-in submitblock backup mirroring the DGB reference (#82): - btc/coin/rpc_conf.hpp: header-only bitcoin.conf credential parser (twin of dgb/coin/rpc_conf.hpp). rpcpassword stays off the process table; --coin-rpc carries HOST:PORT only, --coin-rpc-auth points at the conf file. - btc::coin::Node::arm_submit_rpc(): constructs+connects NodeRPC for submitblock ONLY (no getwork side effect), plus has_rpc(). - main_btc: load creds (default ~/.bitcoin/bitcoin.conf), arm when armed()==true, otherwise stay UNARMED (daemonless default, byte-identical to before). - rpc_conf_test.cpp: 5 KATs (armed/unarmed/missing-file/aliases/endpoint-override). Broadcast-config path only -- no PoW hash, share format, coinbase commitment, or PPLNS math is touched (consensus-neutral). BTC-fenced; no shared/core edit. Builds clean (c2pool-btc + btc_share_test); KATs pass. * btc: harden ARM B submitblock backup — genesis, parser, throw-guard, socket deadline (#744/#787) Arming ARM B (m_rpc) made four previously-dead defects live on the reward path. Review (review) findings, all fixed: B1 (blocker) — NodeRPC::check() probed the LITECOIN genesis hash (copied from the LTC impl, a #759-class cross-coin drift). On BTC mainnet the is_main_chain && !has_block gate always failed -> permanent 15s reconnect loop, ARM B degraded on the one net where a lost block is real money. New btc/coin/genesis.hpp + btc_genesis_hash(IS_TESTNET); check() now probes the per-net BTC genesis (mirrors the DGB reference). genesis_check_test.cpp pins it (regression guard vs the LTC hash). B2 (blocker) — unguarded std::stoi in rpc_conf.hpp aborted the node at startup on a malformed rpcport (e.g. rpcport=abc) in the DEFAULT ~/.bitcoin/bitcoin.conf, crashing operators who never opted in. New conf_detail::parse_port (digits-only, range-checked, never throws) -> junk degrades to UNARMED. 3 new KATs. M1 (major) — a daemon unreachable at submit raised JsonRpcException out of submit_block_hex; broadcast_block_for_connect had no throw guards (unlike the core SSOT), so the exception destroyed ARM A's already-succeeded relay -> false 'reached NEITHER -- lost subsidy' alarm for a block that WAS relayed. Both legs now guarded. 3 new throwing-leg KATs. M2 (major) — no socket deadline on the beast tcp_stream shared with stratum on one single-threaded ioc; a hung bitcoind froze the whole node mid-won-block. Added apply_socket_timeouts() (SO_SND/RCVTIMEO, mirrors DASH #781), armed after each (re)connect. Also softened the arm_submit_rpc comment (check() does issue read-only probes). Broadcast/RPC-transport path only -- no PoW, share, coinbase-commitment, or PPLNS math touched (consensus-neutral). BTC-fenced. Builds clean (c2pool-btc + btc_share_test); 20 KATs pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The DASH mining-hotel primary node stalled stratum under load (~66 connections from ~23 rigs × 2 pools → ~10 rigs could not (re)connect, 0 hashrate). Root cause:
main_dash.cppran theStratumServer, coin-P2P, the blockingNodeRPC, and the 3 s tip-poll's blockinggetbestblockhashall on oneio_contextthread, so the io thread saturated and marginal stratum sessions starved. Plus a zombie-session leak kept dead NAT connections alive forever, inflating the session count.This ports LTC's proven decouple (
thread_pool(1)+ post-back-to-ioc,main_ltc.cpp) into DASH and adds live-session hygiene. Threading/transport only — consensus-neutral (theselect_work/payment logic is moved byte-identical intoresource_template_now(); consensus diff-grep empty).What changed (F1–F3)
F1 — zombie-session leak (the acute N-collapse), core
StratumServer, all gated by newStratumConfigknobs that DEFAULT OFF so LTC/BTC/DGB are byte-unchanged; DASH opts in:is_connected()(socket_.is_open()) never falsifies for a half-open NAT drop, so the kernel now probes and errors the read → normal disconnect+prune.mining.authorize.F2 — the structural decouple (
main_dash.cpp+dash/coin/rpc.{hpp,cpp}+dash/stratum/work_source.{hpp,cpp}):rpc_poolruns the fallback arm's blockinggetbestblockhash; the tip-change follow-up (invalidate_template_cache+bump_work_generation+notify_all) is posted back ontoioc. Timer re-armed only after the RPC returns (no poll pile-up). Lost-block prevention preserved.NodeRPC::Sendserialized withm_rpc_mutex(io-threadgetworkvs pool-threadgetbestblockhashshare one beast client) + a kernelSO_SND/RCVTIMEOdeadline so a wedged dashd can't hang the pool thread.cached_work()serves the cached template immediately and hands the blocking re-source torpc_poolas a single-flight background job → a per-share generation bump no longer blocks the io thread on a full GBT.rpc_poolisstop()+join()ed before its referents unwind at shutdown (bounded by the RPC timeout).F3 (notify_all): left as-is — its sends are already async (
async_write), it freezesbest_shareonce per cycle, and it only fires on a genuine tip change (~per DASH block). The acute per-share/per-3 s io blocking is removed by F2. Deeper per-session-handler batching deferred (higher risk on a live node).Tests / build
DashStratumWorkSource.IoThreadDecoupleServesCachedAndRefreshesOffThread(in the allowlistedtest_dash_stratum_work_source): pins that with an executor wired, a cache-miss / gen-bump does not call the blocking fallback on the io thread, schedules exactly one single-flight background refresh, and runs the re-source off-thread.test_dash_stratum_work_sourcesuite: 34/34 pass.c2pool-dashbuilds clean (boost 1.90); packagedv0.2.3.8self-reportsc2pool-dash 0.2.3.8.Consensus-neutrality
git diffgrep forget_share_bits|share_target|share_bits|DGW|difficulty|oracle|payout|pplns|coinbase_value|masternode|superblock|target_from_nbits|select_work|weightsover added code lines → empty.