btc(stratum): memoize tx_data across work-gen calls (H5 leak fix) - #119
Conversation
build_connection_coinbase rebuilt the per-job tx hex vector on every call
even when the mempool tx set was unchanged. The a1 fix collapsed the
per-job COPIES into shared_ptr refcount bumps, but the vector itself was
still freshly materialized each call -- the dominant churn/leak site in
the 2026-06-02 heaptrack (work_source.cpp:634, 768.18 MB / 1.2M calls,
~78%% of total leaked bytes, linear/unbounded over the trace).
Add a single-slot memo keyed to a fingerprint computed over wd->m_hashes
(the merkle leaf set, in merkle-leaf order) so a repeat call against the
same tx set returns the cached shared_ptr instead of re-serializing the
whole mempool. The fingerprint keys on the exact leaves that built this
job merkle, so a memo hit is atomic-with-merkle by construction (no
stale-tx-vs-fresh-merkle mismatch on submit). Single slot => bounded
memory (one tx set retained), self-evicting on tx-set roll.
BTC-local: touches only src/impl/btc/stratum/work_source.{cpp,hpp}; tx_data
is populated at this one impl site, core only plumbs it, all other coins
carry null -- no cross-coin entanglement.
ctest: 60/60 BTC stratum/template/mining-interface tests green.
|
Methodology correction (supersedes the f929f9e1 arena-frag / churn-not-retention read — do not close this as an instrumentation artifact): heaptrack_print on a live-growing .zst only emits the MOST-CALLS-TO-ALLOCATION section; the PEAK and LEAKED rankings require a second pass over a closed/static stream. The earlier 0B-peak top sites were an artifact of reading the growing stream, not evidence of churn. A static-copy pass over the 2026-06-02 trace shows leaked ~1.03 GB ≈ peak ~1.05 GB, with arena-fragmentation residual only ~20 MB. That is retention, not churn — a real tracked-heap leak. Dominant site is work_source.cpp:634 (build_connection_coinbase tx-hex vector, 768.18 MB / 1.2M calls, ~78%% of leaked bytes, linear/unbounded over the trace), which is exactly what the memoize fix in b000c32 targets. So this fix is warranted and should not be withdrawn as arena-frag. Open question being characterized this heartbeat: whether the remaining ~22%% is a second retention site (mining_submit json get<>) needing its own patch, or also collapses under the b000c32 memo. Will update with the PEAK/LEAKED #2-site ranking. |
|
Correction — withdraw the arena-fragmentation read (was msgid f929f9e1). Methodology note for the record: The retention concentrates at the per-call tx_data rebuild in |
|
Review verdict (integrator): APPROVE for merge. Verified at b000c32:
Supersedes #117 a2 on the live path. Merge gated on operator tap (live-path fix, not SAFE-COSMETIC) — surfacing now. Merge with NO --delete-branch per current rule. |
BTC H5 leak fix — memoize
tx_dataacross work-gen callsLeak site:
src/impl/btc/stratum/work_source.cpp:634—build_connection_coinbasere-materialized the per-job transaction hex vector on every call, even when the mempool tx set was unchanged.Before (2026-06-02 fixture,
print-2026-06-02.txt)work_source.cpp:634leaked 768.18 MB / 1.2M calls = ~96% of all leaked string bytes, 78% of total leaked bytes, linear/unbounded over the ~1h50m trace.tx_dataduplication, NOT collapsed by a1.Fix
Single-slot memo keyed to a fingerprint over
wd->m_hashes(the merkle leaf set). On a repeat call against the same tx set, the cachedshared_ptris returned (refcount bump) instead of re-serializing the whole mempool.Two confirmations integrator asked for
CHash256(double-SHA256) overwd->m_hashesiterated in merkle-leaf order (the vector order), each leaf fed as its 32 raw bytes. Same tx set => same leaf order => same fingerprint, across calls. Because a txid IS the hash of its tx, identicalm_hashes=> identicaltx_data, so the key uniquely determines the cached value (no collision-driven mismatch).tx_data_fp_+tx_data_memo_). A new fingerprint overwrites the prior entry, dropping its refcount — at most one tx set is retained. No unbounded map; we never need more than the latest tx set.Atomic-with-merkle
The key is the exact leaf set that produced this job's merkle / witness_commitment (captured at
stratum_server.cpp:1395). A memo hit therefore implies an identical merkle — no stale-tx-vs-fresh-merkle mismatch on submit. This is why the key is the tx-set fingerprint, notwork_generation_: on BTCwork_generation_bumps tip-only (main_btc.cpp:1212/1215, no mempool-arrival subscriber) whilebuild_templatereads livemempool_per call, so a work-gen-keyed memo could serve staletx_dataagainst a fresh merkle.a504426 / a1 reconciliation
a504426bis an ancestor of this branch base (1040edec, merge of [dash][S4-pre] payout-credit foundation — advance block.m_txs (leaf, single-coin) #114).shared_ptrconversion atwork_source.cpp:637(per-job COPIES -> refcount bumps) and theactive_jobs_MAX_ACTIVE_JOBS=256bound atstratum_server.cpp:1375are both present on the base.Tests
ctest: 60/60 BTC stratum / template-builder / mining-interface tests pass (100%). (The 9 Dash X11/DGW KAT entries reported by a fullctestrun are***Not Run— their executables are not built in this incremental build dir; out of the BTC lane and not affected by this BTC-local change.)Acceptance gate — post-patch heaptrack (IN FLIGHT)
Per the task, the acceptance gate is a post-patch heaptrack on the 2026-06-02 fixture: the retention curve at
work_source.cpp:634must flatten. Queued next heartbeat (30–60 min run); before/after curve will be appended to this PR before merge.Scope / safety
BTC-local: only
src/impl/btc/stratum/work_source.{cpp,hpp}. Zero core edits, zero cross-coin footprint (tx_datapopulated at this one impl site; all other coins carry null). Commit signedb000c323. Push held — merge operator-gated.