fix: v32 — post-fast-sync catch-up + HBC admission + cache invariant

v32.1 — fast_sync re-verifies network tip before declaring complete.
Reads BEST_PEER_HEIGHT and live get_max_peer_height() each iteration and
takes max(); refuses to break the bulk loop while the actual peer-quorum
tip is still ahead. Prevents premature exit when the network advances
during a batch download.

v32.2 — chronic_stall handler uses adaptive bulk catch-up. When the gap
to peer-quorum tip exceeds 90 blocks, it requests sync_blocks(next,
next + min(gap, 1000)) instead of the fixed 90-block tip-recovery window.
Catch-up rate now scales with the gap rather than capping at 90 blocks
per cooldown.

v32.3 — production gate drives catch-up. When the per-slot production
gate blocks due to !node_synced and the quorum tip is more than 50 blocks
ahead, the gate sets CHRONIC_STALL_REQUESTED so the next chronic_stall
iteration triggers v32.2 bulk catch-up. 30 s per-trigger cooldown
prevents request storms.

v32.4 — admission no longer whitelists HBC sample signature formats. The
v31.7 prefix whitelist did not include the real producer-side format
(hybrid_p2p_bin), so every HeartbeatCommitment TX was rejected on
mempool admission and no HBC ever landed on-chain. Admission now keeps
only structural/DoS checks (size, merkle-proof depth, hex format);
signature-format dispatch and cryptographic verification stay in the
verify-stage path (verify_consensus_signature) and at the eligibility
gate (Phase 2A merkle replay + Dilithium verify).

v32.5 — cache_block_hash is called only after save_microblock succeeds,
not at verify success. The verify-stage cache write let uncommitted
view-change candidate hashes leak into the RAM cache; when apply
selected a different canonical block the cache no longer matched
RocksDB, and the next height's parent-hash check raised a false
hash_chain_break that wedged the verify pipeline (observed at h=25015
after 9 view-change rounds, with a 25 875 s verify_stuck on
verify:load_prev_block). Invariant after this commit:
RECENT_BLOCK_HASHES[h] == storage[h] for every cached h. The RocksDB-hit
backfill at the verify fallback is canonical by construction and is
retained.

Validation: cargo check + lib tests across qnet-integration (147/147),
qnet-consensus (73/73), qnet-state (32/32). 252 tests pass, 0
regressions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: AIQnetLab

core/qnet-state/src/transaction.rs

@@ -1339,39 +1339,29 @@ impl Transaction {
                     }
                 }
 
-                // Validate each sample (v31.7: structural cryptographic bounds).
-                // Chain-state binding (sample.block_height must reference a real
-                // local block) is enforced at the eligibility-gate, not here,
-                // because mempool admission has no storage handle.
+                // Structural sample sanity. Crypto verification (Dilithium sig
+                // + merkle replay vs merkle_root) runs at the eligibility gate
+                // where the chain-state handle is available; admission must
+                // not whitelist signature formats — that job belongs to the
+                // verify-stage dispatcher.
                 for sample in heartbeat_samples {
                     if sample.heartbeat_index >= 10 {
                         return Err(format!("[REJECT][TX] invalid_heartbeat_index value={}", sample.heartbeat_index));
                     }
                     if sample.block_height < *window_start_height || sample.block_height > *window_end_height {
                         return Err(format!("[REJECT][TX] heartbeat_sample_outside_window block_height={} start={} end={}", sample.block_height, window_start_height, window_end_height));
                     }
-                    // v31.7: signature must look like a hybrid Dilithium3+Ed25519
-                    // payload; same prefix/length envelope as verify_consensus_signature.
-                    if sample.signature.len() < 100 || sample.signature.len() > 18_000 {
+                    // DoS-bound only: payload envelope ~5KB; cap generous.
+                    if sample.signature.len() < 64 || sample.signature.len() > 32_768 {
                         return Err(format!(
                             "[REJECT][TX] heartbeat_sample_signature_size len={}",
                             sample.signature.len()
                         ));
                     }
-                    let sig_prefix_ok = sample.signature.starts_with("hybrid:")
-                        || sample.signature.starts_with("hybrid_bin:")
-                        || sample.signature.starts_with("compact:")
-                        || sample.signature.starts_with("compact_bin:")
-                        || sample.signature.starts_with("dilithium_sig_");
-                    if !sig_prefix_ok {
-                        return Err("[REJECT][TX] heartbeat_sample_signature_prefix_unknown".to_string());
-                    }
                     if sample.merkle_proof.is_empty() {
                         return Err("[REJECT][TX] heartbeat_sample_missing_merkle_proof".to_string());
                     }
-                    // v31.7: merkle proof depth bounded — 10 leaves cap at log2(10)≈4,
-                    // accept up to 16 to allow future heartbeat-tree growth without
-                    // letting unbounded proofs inflate TX size.
+                    // 10 leaves ⇒ tree depth ≤ 4; cap at 16 for future growth.
                     if sample.merkle_proof.len() > 16 {
                         return Err(format!(
                             "[REJECT][TX] heartbeat_sample_merkle_proof_too_deep depth={}",

development/qnet-integration/src/block_pipeline.rs

@@ -2020,12 +2020,10 @@ impl BlockPipeline {
                 }
             }
 
-            // All checks passed — forward to apply stage
-            let block_height = decoded.height; // Copy before move
-            // v31.1: cache verified hash now so verify(h+1) hits RAM before
-            // apply finishes writing h. Hash is content-deterministic ⇒ safe.
-            let verified_hash = decoded.microblock.hash();
-            cache_block_hash(block_height, verified_hash);
+            // All checks passed — forward to apply stage.
+            // v32.5: cache populated only on apply-commit, never at verify —
+            // uncommitted view-change candidates must not poison the RAM cache.
+            let block_height = decoded.height;
 
             let verified = VerifiedBlock {
                 height: block_height,
@@ -2381,6 +2379,10 @@ impl BlockPipeline {
                             &block.microblock,
                         );
 
+                        // v32.5: publish canonical parent-hash to RAM cache only
+                        // after RocksDB commit — invariant cache == storage.
+                        cache_block_hash(height, block.microblock.hash());
+
                         // ═══════════════════════════════════════════════════════
                         // v25 H9: VALIDATOR LIVENESS — SUCCESS PATH
                         // ───────────────────────────────────────────────────────

development/qnet-integration/src/node.rs

@@ -17447,8 +17447,22 @@ impl BlockchainNode {
                                     let _ = p2p.sync_macroblocks(
                                         missing_mb.saturating_sub(1), latest_mb,
                                     ).await;
-                                    let resync_from = next_height.saturating_sub(90);
-                                    let _ = p2p.sync_blocks(resync_from, next_height).await;
+                                    // v32.2: adaptive resync range. If gap to network tip > 90,
+                                    // request bulk forward range (up to 1000 blocks) instead of
+                                    // fixed backward 90-block tip window. Tip recovery only when close.
+                                    const BULK_CATCHUP_THRESHOLD: u64 = 90;
+                                    const BULK_CATCHUP_CHUNK: u64 = 1000;
+                                    let quorum_peak = p2p.get_max_peer_height();
+                                    let gap_to_tip = quorum_peak.saturating_sub(next_height);
+                                    if gap_to_tip > BULK_CATCHUP_THRESHOLD {
+                                        let bulk_to = next_height.saturating_add(gap_to_tip.min(BULK_CATCHUP_CHUNK));
+                                        println!("[INFO][SYNC] chronic_stall bulk_catchup from={} to={} gap={}",
+                                                 next_height, bulk_to, gap_to_tip);
+                                        let _ = p2p.sync_blocks(next_height, bulk_to).await;
+                                    } else {
+                                        let resync_from = next_height.saturating_sub(90);
+                                        let _ = p2p.sync_blocks(resync_from, next_height).await;
+                                    }
                                 }
                             }
                         }
@@ -18077,12 +18091,20 @@ impl BlockchainNode {
                                                 }
                                                 drop(global_h);
 
-                                                // Check if we've caught up (use lowered threshold matching fast sync trigger)
-                                                let new_target = crate::unified_p2p::BEST_PEER_HEIGHT.load(Ordering::Relaxed);
+                                                // v32.1: re-verify network tip via authoritative peer-quorum height
+                                                // before declaring sync complete. BEST_PEER_HEIGHT alone can lag
+                                                // attestation TTL; pull live quorum max so we don't exit early.
+                                                let best_atomic = crate::unified_p2p::BEST_PEER_HEIGHT.load(Ordering::Relaxed);
+                                                let quorum_max = p2p_clone.get_max_peer_height();
+                                                let new_target = best_atomic.max(quorum_max);
                                                 if current_from + 3 >= new_target {
-                                                    println!("[INFO][SYNC] fast_sync_complete h={} network={}", current_from, new_target);
+                                                    println!("[INFO][SYNC] fast_sync_complete h={} network={} (quorum={})",
+                                                             current_from, new_target, quorum_max);
                                                     break;
                                                 }
+                                                if new_target > best_atomic {
+                                                    crate::unified_p2p::BEST_PEER_HEIGHT.store(new_target, Ordering::Release);
+                                                }
 
                                                 // Advance from for next batch
                                                 current_from += 1;
@@ -18641,9 +18663,11 @@ impl BlockchainNode {
                     }
                 }
 
-                // v11.0: HARD GATE — no production while sync in progress
-                // Prevents producing blocks at stale height while sync is still running.
-                // Node must be fully synchronized AND production unlocked (first network block received).
+                // v11.0+v32.3: HARD GATE — no production while not synced.
+                // v32.3 addition: if local is far behind quorum peer max, trigger
+                // CHRONIC_STALL_REQUESTED so the bulk catch-up handler engages on
+                // next iteration. Without this, the gate just blocks forever
+                // waiting for sync that no path is driving.
                 if is_my_turn_to_produce {
                     let sync_active = coordinator_is_syncing();
                     let prod_unlocked = PRODUCTION_UNLOCKED.load(Ordering::Relaxed) == 1;
@@ -18655,6 +18679,32 @@ impl BlockchainNode {
                                      next_block_height, sync_active, prod_unlocked, node_synced);
                         }
                         is_my_turn_to_produce = false;
+
+                        // v32.3: when blocked due to !node_synced, drive bulk catch-up.
+                        if !node_synced && !sync_active {
+                            if let Some(ref p2p) = unified_p2p {
+                                let local_h = next_block_height.saturating_sub(1);
+                                let quorum_peak = p2p.get_max_peer_height();
+                                const PROD_GATE_BULK_GAP: u64 = 50;
+                                if quorum_peak > local_h + PROD_GATE_BULK_GAP {
+                                    static LAST_GATE_BULK_TRIGGER: std::sync::atomic::AtomicU64 =
+                                        std::sync::atomic::AtomicU64::new(0);
+                                    const GATE_BULK_COOLDOWN_SECS: u64 = 30;
+                                    let now_u64 = std::time::SystemTime::now()
+                                        .duration_since(std::time::UNIX_EPOCH)
+                                        .map(|d| d.as_secs()).unwrap_or(0);
+                                    let last = LAST_GATE_BULK_TRIGGER.load(Ordering::Relaxed);
+                                    if now_u64.saturating_sub(last) >= GATE_BULK_COOLDOWN_SECS {
+                                        LAST_GATE_BULK_TRIGGER.store(now_u64, Ordering::Relaxed);
+                                        CHRONIC_STALL_REQUESTED.store(true, Ordering::Relaxed);
+                                        if is_info() {
+                                            println!("[INFO][PROD] catchup_requested local={} quorum={} gap={}",
+                                                     local_h, quorum_peak, quorum_peak - local_h);
+                                        }
+                                    }
+                                }
+                            }
+                        }
                     }
                 }