dash(embedded): mainnet-validity — real-dashd CbTx byte-parity + C-3/H-4/superblock (v0.2.4 gate-lift proof)

[frstrtr]

The GATE-LIFT proof for the interim mainnet gate carried by #779. Branches off #779, so it includes the C-2/H-6/H-7 SML-axis wiring; this PR adds what actually lets the mainnet gate lift and v0.2.4 ship.

THE deliverable: real-dashd byte-parity (H-5)

The prior embedded_gbt KAT was self-referential (c2pool built both sides). This replaces it with a verbatim Dash Core testnet getblocktemplate capture (block 1,518,413) as the oracle — test/fixtures/dash_testnet_gbt_1518413.json. The coinbase_payload is dashd's own type-5 CbTx extra_payload, not c2pool output.

Result (test_dash_embedded_cbtx_byte_parity.cpp, 4/4 green):

• parse_cbtx -> encode_cbtx reproduces dashd's 175-byte CbTx payload BYTE-IDENTICALLY — every field: nVersion, nHeight, merkleRootMNList, merkleRootQuorums, VarInt bestCLHeightDiff, the 96-byte BLS bestCLSignature, and the signed int64 creditPoolBalance.
• build_embedded_cbtx assembly reproduces dashd's non-root fields exactly (v3, nHeight=prev+1, bestCLHeightDiff = prev − bestCLHeight = 1, verbatim ChainLock sig, creditPool); with dashd's roots transplanted, the whole payload is byte-identical.

No field diverged. The one piece not recomputed from scratch in-KAT is deriving the two merkle roots from the specific 375-entry / 109-quorum testnet SML (a heavy wire rebuild); those roots are proven equal to dashd's commitment out-of-band (the captured protx diff 1 1518412 reports the same roots dashd puts in the GBT) and root computation is pinned by test_dash_simplifiedmns / test_dash_quorum_root.

Gaps closed by WIRING (not recoding)

• C-3 special-tx filter — get_sorted_txs_with_fees gains an exclude_special option; build_embedded_workdata passes true so embedded templates carry no special tx (dashd recomputes CbTx roots + creditPool from the block's own special txs → an unaccounted one = bad-cbtx). Default false preserves the mempool's general pricing (asset-unlock fee path unchanged).
• H-4 creditPool accrual — creditPoolBalance(N) = creditPoolBalance(N-1) + platformReward(N). Verified byte-exact against dashd's own consecutive-block series (every testnet block stepped the pool by exactly 66966830 duff). The prior code committed the stale last-observed balance — off by one platform reward.
• superblock guard — on a superblock-height next block the embedded arm refuses (routes to the reward-safe dashd fallback that carries the governance outputs). Network-aware cycle (mainnet 16616, testnet 24).

Verdict on mainnet daemonless validity

The CbTx wire form, field assembly, bestCL derivation, and creditPool accrual are proven byte-identical to a real dashd. Remaining before a mainnet gate-lift:

1. reconstruct the two roots from a raw mnlistdiff wire capture and assert CalcMerkleRoot == fixture root (roots already proven == dashd's commitment; this closes the last from-wire step);
2. subsidy.hpp uses mainnet activation heights — fine for the mainnet gate target, but testnet embedded mining needs per-net heights before it is byte-correct (flagged, not a mainnet blocker).

Once (1) lands, lifting the interim mainnet gate in stratum/work_source.cpp is a small follow-up — the v0.2.4 trigger.

Draft — gate-lift evidence for review.

[frstrtr]

From-wire merkle-root parity — result per root

Closed the last open link: computed both CbTx merkle roots from a raw testnet mnlistdiff P2P capture (getmnlistd(base=ZERO, block=1518412), committed as test/fixtures/dash_testnet_mnlistdiff_1518412.bin, 97639 bytes, checksum-verified off the socket — dashd's bytes, not ours) and compared to dashd's committed roots.

merkleRootMNList — MATCHES ✅
raw mnlistdiff → vendor apply_diff into an empty SML → CalcMerkleRoot == 6bbc07fd…48f, byte-identical to dashd. The SML axis is now proven end-to-end from wire, no longer transplanted.

merkleRootQuorums — DIVERGES ❌ (gate-lift blocker)
Ours e839698c… vs dashd 1901c172…. Precisely isolated — it is only the leaf flatten-ordering in compute_merkle_root_quorums:

• active quorum set exact — 109, per-type 24/4/1/24/32/24, identical to protx diff 1 1518412;
• wire delivery order identical to dashd's protx-diff order;
• leaf hash byte-exact — pack(commitment) reproduces the wire bytes verbatim (found in the tail), so SHA256d(pack) == dashd's SerializeHash(qc);
• merkle machinery is the same helper that reproduces merkleRootMNList.

No wire-derivable ordering tried (wire order; per-(llmqType,quorumHash) LE/BE; global quorumHash; per-type leaf-hash sort; global leaf-hash sort) reproduces dashd's root. dashcore's CalcCbTxMerkleRootQuorums orders via GetMinedAndActiveCommitmentsUntilBlock (mining-height driven); mining_height is not carried by the mnlistdiff wire, so the daemonless quorum path cannot currently reconstruct dashd's order. The exact flatten order needs the dashcore reference — not guessed here.

Tracked by DISABLED_MerkleRootQuorumsFromWireMatchesDashd (fails with the exact values when enabled); CI stays green.

Corrected verdict

The interim mainnet gate must NOT lift yet. merkleRootMNList, the full CbTx wire form, bestCL, and creditPool accrual are all proven byte-identical to real dashd — but a wrong merkleRootQuorums = bad-cbtx (lost subsidy). The single remaining blocker is the quorum-root flatten-order fix (either track mining_height from the qc-commitment stream and order newest-first per type, or confirm the exact dashcore leaf order against the reference). No gate-lift PR until that root matches from wire.

[frstrtr]

merkleRootQuorums RESOLVED — both roots match dashd from wire; gate lifted

Read dashcore's CalcCbTxMerkleRootQuorums directly: it collects ::SerializeHash for every mined+active commitment and does a final std::sort of the leaf hashes before the merkle. So ordering and mining-height are irrelevant — only the SET matters, and it is fully wire-derivable (no block-body qc ingest, no mining_height needed).

Root cause: compute_merkle_root_quorums was dedup'ing rotated types (5/6) by quorumIndex, dropping ring-member leaves — its set was smaller than dashd's 109. An independent recompute over our exact 109 leaf hashes (sort + merkle) reproduced dashd's 1901c172… exactly, confirming leaves+machinery were already correct.

Fix: hash all active commitments (no per-index dedup, no mining-height ordering), sort the leaf hashes (memcmp), merkle — matching dashcore.

Per-root from-wire result (raw testnet mnlistdiff → our objects → our roots)

• merkleRootMNList == dashd 6bbc07fd…48f ✅
• merkleRootQuorums == dashd 1901c172…dfb ✅ (was e839698c…)

test_dash_mnlistdiff_root_parity: MerkleRootQuorumsFromWireMatchesDashd now enabled and passing. test_dash_quorum_root updated — RotatedRingMembersAllIncluded replaces the incorrect dedup KAT; the self-referential golden was re-pinned to the corrected all-inclusive set (the real consensus anchor is now the from-wire dashd parity, not a self-golden).

Gate-lift (v0.2.4 trigger)

DASHWorkSource::set_embedded_mainnet + --embedded-mainnet enable the daemonless embedded arm on mainnet, now that the full CbTx (wire form, both roots, bestCL, creditPool accrual) is proven byte-identical to real dashd. Default OFF keeps unconfigured mainnet nodes fail-closed on the reward-safe dashd fallback; even when enabled, the NodeCoinState viability gate (SML+quorum fresh at tip, non-superblock) fails safe to the fallback, and [GBT-XCHECK] remains.

Verdict

Mainnet daemonless CbTx is now provably byte-identical to dashd end-to-end from the wire. The gate can lift (opt-in). Suites green: quorum_root 11, mnlistdiff_root_parity 3, byte_parity 4, embedded_gbt 9, stratum_work_source 35, node_coin_state 9; c2pool-dash builds+links. Draft — no merge.

[frstrtr]

Review addressed — three fail-closed guards land; --embedded-mainnet=ON is now reward-safe

Thanks for the adversarial review — both consensus holes confirmed and fixed. ON now fails closed (worst case = dashd fallback, never a bad block). Commit 27f8270b.

BLOCKER-1 (CRITICAL) — DKG mining-phase heights — PHASE 1 shipped (fail-closed).
is_dkg_commitment_window(height) replicates dashcore IsMiningPhase over every enabled llmqType's [dkgMiningWindowStart,End] (params verbatim from llmq/params.h: (24,10,18) for 50_60/100_67/25_67, (288,20,28) 400_60, (576,20,48) 400_85, (288,42,50) 60_75 — union is network-agnostic + fail-safe). Wired into NodeCoinState viability → those heights route to the dashd fallback. KAT proves the live commitment heights 1518418/1518419/1518442 fail closed and the fixture 1518413 proceeds.
Phase 2 (daemonless qc sourcing) tractability: NOT a small follow-up. Minimal mining (relay-not-verify) needs a qfcommit/qgetdata P2P handler + a special-tx mempool lane (currently stripped by C-3) + qc→merkleRootQuorums + creditPool re-derivation — a real slice, no BLS strictly required to relay. Independent validation of commitments is Phase-L/BLS-blocked (no BLS lib linked; quorumsCLSigs kept verbatim for Phase L). Recommend Phase-1 fail-closed as the shipped v0.2.4 behavior; Phase-2 as a milestone.

BLOCKER-2 (HIGH) — bestCL staleness — fixed (fail-closed).
require_fresh_bestcl viability gate: best_cl_height >= prev_height - 1. Sufficient by construction — the previous block's committed CL height is <= prev_height-1, so a CL that fresh is guaranteed non-null and non-regressing (matches the fixture: block 1518412 committed CL height 1518411 = prev-1). Post-restart / relay-gap → fail closed. KAT.

BLOCKER-3 — pre-emit hard gate — done.
NodeCoinState::embedded_template_emit_ok re-validates the BUILT CbTx before serve: re-parse; nHeight==prev+1; both merkle roots re-derived from the current SML/QuorumManager match the committed roots; height-class + bestCL guards re-asserted. Wired into the DASHWorkSource serve path — a failing embedded template is discarded for the dashd fallback. The "safety path" comment is now true on the hot path. KAT (valid accept; tampered-root / empty-payload / commitment-window reject).

Nit — quorum-tail health. The maintainer now signals set_quorum_healthy(parse_ok); a malformed quorum tail (stale QuorumManager while SML advances) fails the arm closed until a clean diff lands.

All guards default OFF/unset (default-OFF path byte-unchanged); main_dash arms them when the arm can serve (testnet || --embedded-mainnet). Suites green: node_coin_state 13, coin_state_maintainer 9, stratum_work_source 35, embedded_gbt 9, quorum_root 11, mnlistdiff_root_parity 3, byte_parity 4; c2pool-dash builds+links.

Soak plan (before any merge/tag)

Testnet, --embedded-mainnet ON on the testbed, across the risky boundaries, asserting 0 bad-cbtx / 0 bad-qc:

1. DKG interval crossing — run through several %24 ∈ [10,18] windows (e.g. 1518418/19/42-class heights): expect the served-arm log to show dashd-fallback at every commitment-window height and EMBEDDED on clear heights; 0 embedded templates at qc heights.
2. Superblock boundary — cross a testnet superblock height (cycle 24): fallback at the boundary.
3. Restart-with-stale-CL — restart the node and confirm the arm stays on fallback until a fresh clsig lands (bestCL gate), then flips to EMBEDDED.
4. Pre-emit gate — confirm no pre-emit check FAILED lines accompany an emitted block (any failure = discarded → fallback).
   Success = across a full DKG interval + a superblock + a restart, every emitted block validated by the testbed dashd (submitblock accept / getblock present), zero bad-cbtx/bad-qc. Then merge + tag v0.2.4 as the genuinely-independent daemon.

Draft — no merge/tag.

[frstrtr]

H-1 fixed (option-b HEAL) + both nits — the last code gate

Commit 6e5599a0. Took the review's preferred option (b): HEAL, not latch.

H-1 — quorum-health un-latch (HIGH): the maintainer now parses the quorum tail first; on failure it treats it as a quorum-axis reorg — on_sml_reorg() wipes SML+qmgr (have_sml=false, sml_current_hash=ZERO) and m_on_full_resync forces a full snapshot from ZERO (main_dash resets the sml_base tracker + re-requests getmnlistd(ZERO,tip)). The diff's SML is not applied — discard + re-sync, so a base-relative delta can never be silently skipped.

Crucially, the base-continuity guard is tightened: while sml_current_hash==ZERO (cold OR post-wipe) only a full snapshot (base=ZERO) is accepted — a clean incremental after a wipe is now rejected, closing the exact un-latch (a later incremental can no longer flip health back true over a permanently-missing delta). State converges instead of latching off.

KAT (test_dash_coin_state_maintainer): full snapshot (ok) → incremental w/ malformed tail (wipe + resync fired, have_sml=false, quorum_healthy=false) → clean INCREMENTAL still refuses (base-continuity rejects, state unchanged) → full snapshot recovers. This is the "malformed tail → clean incremental → must STILL refuse" case you required.

Nit 1 (symmetry): embedded_template_emit_ok now re-asserts m_quorum_healthy.
Nit 2 (maintenance): dkg_window.hpp carries a "re-diff vs llmq/params.h on every vendored-pin bump" note.
Residual (documented, non-blocking): clsig is ingested without BLS verify (Phase L) — a malicious direct peer could inject a bogus fresh (height,sig) → one lost block (bad-cbtx-invalid-clsig), not chain acceptance / no reward theft. Recorded in the writeup as a Phase-L follow-on; mitigation is trusted coin-P2P peers under ON. Doesn't block the soak.

Suites green: coin_state_maintainer 10, node_coin_state 13, stratum_work_source 35, embedded_gbt 9, mnlistdiff_root_parity 3, byte_parity 4, node_reception_wire 21; c2pool-dash builds+links. Writeup updated (frstrtr/the p2poolv36). Draft — no merge/tag. This is the last code gate; the parallel soak's results + this fix clear the path to merge+tag.

[frstrtr]

Soak-found bad-cbtx fixed — creditPool freshness gate (4th CbTx axis)

The v0.2.4 soak on 6e5599a0 caught a real reward-losing defect the unit tests + prior reviews missed: bad-cbtx-assetlocked-amount on 2/5 embedded serves (heights 1518558, 1518560), committed creditPoolBalance low by exactly one platform reward (66,966,830 duffs). Fixed in 88788a0b.

Root cause (confirmed against the soak jsonl): the require_sml freshness gate only covered the SML/merkleRoot axis (sml_current_hash == prev_hash). The DIP-0027 credit-pool seed rides a separate on_mnlistdiff step (the diff's embedded cbTx) and can lag one block while the SML hash is already at the tip — at tip 1518559 the seed still held creditPool(1518558), so last_observed + platform_reward committed creditPool(1518559) when block 1518560 needed creditPool(1518559)+reward. The BLOCKER-3 pre-emit gate re-derived only roots + bestCL, not creditPool, so it served the bad block.

Fix (all three parts requested):

• (a) track credit_pool_current_hash — set with the balance whenever the seed updates (== diff.blockHash); when the seed step is skipped (non-type-5/parse-fail cbTx) the tag stays stale while the SML advances. require_fresh_credit_pool gates viability on credit_pool_current_hash == prev_hash.
• (b) the pre-emit hard gate (embedded_template_emit_ok) now also fails closed on a lagged credit-pool seed.
• (c) KAT (node_coin_state): an SML-current-but-credit-pool-LAGGED bundle fails closed on BOTH viability and the pre-emit gate (using the exact soak values 33968316593316 @ stale-hash vs the fresh 33968383560146), and serves only when the seed is current at the tip.

on_sml_reorg invalidates the credit-pool freshness too. main_dash arms require_fresh_credit_pool when the arm can serve (testnet || --embedded-mainnet), like the bestCL gate. Default OFF → unit-test posture unchanged.

The intermittent nature (3/5 serves were correct) means the gate fails-closes exactly the lagged cases and still serves when the seed is fresh — reward-safe without killing the arm.

Suites green: node_coin_state 14, coin_state_maintainer 10, stratum_work_source 35, embedded_gbt 9, mnlistdiff_root_parity 3, byte_parity 4, node_reception_wire 21; c2pool-dash builds+links. Writeup updated (frstrtr/the p2poolv36). Draft — no merge/tag. Ready for the re-soak on 88788a0b; gate is 0 bad-cbtx.

[frstrtr]

Re-soak failure fixed — creditPool VALUE re-check (hash-proxy was unsound)

The re-soak on 88788a0b correctly refuted the hash-tag proxy (2/10 bad-cbtx, preemit_failed=false). Fixed in bf1a5312 with a value check, not a hash proxy — mirroring how the merkle roots are re-derived.

Root cause: credit_pool_current_hash == prev_hash is a block-hash tag used as a proxy for balance freshness. Under a build-vs-serve caching skew the template is BUILT with a stale credit-pool seed and CACHED; by serve time the hash-tag has gone fresh, but the pre-emit gate re-derived only the two merkle roots, never re-comparing the cached CbTx's baked creditPoolBalance to the current seed. (protx diff confirmed the wire seed was correct — a cache/timing skew, not a bad diff.)

Fix (both, value re-check is load-bearing):

• embedded_template_emit_ok now re-derives expected = current_seed + platform_reward(next_h) (the SAME accrual build_embedded_workdata does) and requires the BUILT CbTx's creditPoolBalance == expected. It's a pure seed-delta check (the reward cancels), so a template built over an older seed fails closed — exactly like the merkle-root re-derivation catches a stale SML.
• cached_work now re-validates an EMBEDDED cache HIT against the current coin-state before serving it (template_cache_is_embedded_ tag + serve-time emit_ok); a stale-built template is dropped and re-sourced. This closes the skew on the actual serve path, not just the predicate.

KATs (build-vs-serve, per your requirement):

• node_coin_state: build over seed S(N-2), advance to S(N-1) keeping the hash-tag fresh (so the OLD gate passes) → the VALUE check REJECTS the stale-built template, accepts a freshly-built one. Real re-soak values (33971546001156 → 33971612967986, +66,966,830 duff).
• stratum_work_source serve-path: a stale-cached embedded template is dropped + re-sourced over the advanced seed at serve time (log: cached EMBEDDED template failed serve-time re-check — re-sourcing).

Suites green: node_coin_state 15, stratum_work_source 36, coin_state_maintainer 10, embedded_gbt 9, mnlistdiff_root_parity 3, byte_parity 4, node_reception_wire 21; c2pool-dash builds+links. Writeup updated (frstrtr/the p2poolv36). Draft — no merge/tag. Ready for re-soak on bf1a5312; gate is 0 bad-cbtx across a DKG interval + superblock + restart.

[frstrtr]

Re-soak #2 fixed — INDEPENDENT seed-height gate + GBT-xcheck (v3, upstream at the seed)

Re-soak #2 correctly refuted the self-referential fixes: the credit-pool seed itself lagged one block after a cold-snapshot restart, and a seed-derived self-check (hash OR value) cannot catch a wrong seed (built = stale_seed + reward is self-consistent but wrong). Fixed in 9df27106.

Diagnosis (traced the seed, verified the wire): captured a real testnet incremental mnlistdiff (base=1518667, blockHash=1518669) and decoded it through the vendor codec — its embedded cbTx carries nHeight=1518669 and creditPool=creditPool(1518669), i.e. the TIP's seed (full-snapshot likewise). So the wire + deserialization + seed code are correct; the live staleness is an async seed-lag, and the fix must be an independent verify that does not derive from the seed value.

Fix v3 (upstream, at the seed):

1. INDEPENDENT seed-height gate. set_credit_pool now records the seed cbTx's OWN nHeight (authoritative off the wire). Viability and the pre-emit gate require credit_pool_height == m_prev_height (the tip). A seed one block behind fails closed regardless of its value/hash — the only check that catches it. Replaces the unsound hash-tag proxy.
2. GBT-xcheck reward-safety BACKSTOP. When a dashd is reachable, cross-check the embedded CbTx creditPool against dashd getblocktemplate before serving; on a same-height mismatch, serve dashd's template. Catches ANY seed bug present/future. Opt-in (set_gbt_xcheck, enabled for testnet || --embedded-mainnet where a dashd fallback exists); pure-daemonless relies on the height gate.

KATs (per your #4 — model the post-restart transition):

• coin_state_maintainer POST-RESTART: cold snapshot @1518654 → first incremental @1518655 must advance the seed (credit_pool_height 1518654→1518655); a non-advancing diff leaves it behind and the height gate fails closed at the next tip. Diffs carry real type-5 cbTx seeds.
• node_coin_state: a seed one block behind the tip refuses embedded (real re-soak values 1518655 vs tip 1518656); the VALUE re-check still rejects built != current seed + reward (defence in depth).
• stratum: GBT-xcheck serves dashd on a creditPool mismatch.
• mnlistdiff_root_parity: real full + incremental wire fixtures prove the diff cbTx carries the seed for its own block.

On the seed-advance (#1): the ingestion code + wire are correct (proven above); the live lag is async timing. The independent height-gate makes it reward-safe (fails closed when the seed's own height is behind the tip; serves when it matches), and the GBT-xcheck is the backstop. If the async lag proves persistent in the re-soak (arm falling back too often rather than serving), that's a throughput tune, not a reward-safety issue.

Suites green: node_coin_state 15, coin_state_maintainer 11, stratum_work_source 37, embedded_gbt 9, mnlistdiff_root_parity 4, byte_parity 4, node_reception_wire 21; c2pool-dash builds+links. Writeup updated (frstrtr/the p2poolv36). Draft — no merge/tag. Ready for re-soak #3 on 9df27106; gate = 0 bad-cbtx across a DKG interval + superblock + restart.

[frstrtr]

C-3 special-tx exclusion — VERIFIED present + applied (not a bug); end-to-end KAT added (8f50a4f9)

Checked the oracle-design critique's claim that special txs can reach the embedded block. The filter is present and applied on the embedded serving path — no source fix needed:

• build_embedded_workdata (the sole populator of the block's w.m_txs) calls get_sorted_txs_with_fees(MAX_BLOCK_BYTES, /*exclude_special=*/true).
• The filter if (exclude_special && entry.tx.type != 0) continue; (mempool.hpp:393, after the fee-known check) drops every non-zero tx type — a superset of the required DIP types 1-4,6,8,9 (type 5 is the coinbase, never a mempool tx).

The critique reconciles to: the default path (exclude_special=false, needed for the general mempool pricing / asset-unlock fee tests) does not filter, but the embedded arm always passes true. So special txs cannot reach the embedded block.

Why it must exclude them: build_embedded_cbtx commits merkleRootMNList / merkleRootQuorums / creditPool from the SML/quorum/seed state without applying the block's own txs' state effects — a selected ProTx (1-4) / qc-commitment (6) / asset-lock-unlock (8/9) would make the committed CbTx inconsistent → bad-cbtx / bad-merkle. Modeling own-block special-tx deltas is a Phase-2 lift.

Added an END-TO-END KAT (previously only the get_sorted_txs_with_fees level was pinned): mempool seeded with a standard type-0 tx + DIP special txs of every type 1,2,3,4,6,8,9 — each UTXO-priced (fee-known) so it would be selected but for the type filter — built via build_embedded_workdata; asserts the block contains only the type-0 tx and no special-tx id leaks in. embedded_gbt 9→10; mempool 19 (existing get_sorted-level C-3 test intact).

Orthogonal to the credit-pool re-soak #3; both clean for v0.2.4. Draft — no merge/tag.