fix: v16.0.2 idempotent state apply (sync re-delivery resilience)

Re-delivery of an already-applied block during catch-up sync fails the
strict nonce check (`expected=N got=K<N`), causing partial block apply,
state divergence, and a permanent hash_chain_break at the next block.

Symptom on the testnet:
  [REJECT][TX] invalid_nonce expected=101 got=1..100
  [REJECT][CREATE-ACCOUNT] account_already_exists
  [WARN][PIPELINE] hash_chain_break h=351 from=genesis_node_001
  [CRIT][PIPELINE] verify_stuck stall_ms=1060000+ verified=271 applied=271

Fix: silent Ok() when tx.nonce <= sender.nonce (already-applied) across
all 9 nonce-checked tx_types in apply_to_state, plus matching idempotency
for CreateAccount (account_exists) and NodeActivation (already_node).
Same policy mirrored into state_db.rs::execute_transaction.

Replay protection preserved — TXs with stale nonce have no incremental
effect because sender's balance and state already reflect the original
deduction. Strict +1 check still applies for new (>current) nonces.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: AIQnetLab

core/qnet-state/src/state_db.rs

@@ -86,10 +86,19 @@ impl StateDB {
                 }
             });
 
+            // IDEMPOTENT APPLY — silent skip for already-applied transactions.
+            // Mirrors the policy in `transaction.rs::apply_to_state` (Transfer arm)
+            // so that re-delivery of the same TX during sync / replay does not
+            // fail the operation. Replay protection is preserved: a TX with stale
+            // nonce has no incremental effect (sender's balance already reflects
+            // the original deduction).
+            if tx.nonce <= sender.nonce {
+                return Ok(tx_hash);
+            }
             // Check nonce for transaction ordering
             if tx.nonce != sender.nonce + 1 {
                 return Err(StateError::InvalidTransaction(format!(
-                    "Invalid nonce: expected {}, got {}", 
+                    "Invalid nonce: expected {}, got {}",
                     sender.nonce + 1, tx.nonce
                 )));
             }

core/qnet-state/src/transaction.rs

@@ -1541,7 +1541,50 @@ impl Transaction {
                 // Get sender account
                 let sender = accounts.get_mut(from)
                     .ok_or_else(|| StateError::AccountNotFound(from.clone()))?;
-                
+
+                // ═══════════════════════════════════════════════════════════════
+                // IDEMPOTENT APPLY — silent skip for already-applied transactions
+                // ═══════════════════════════════════════════════════════════════
+                // When a node receives the same block more than once (typical during
+                // batch sync, gossip duplication, or post-restart replay), every TX
+                // inside that block is presented to apply_to_state again. Without
+                // idempotency the strict `nonce == sender.nonce + 1` check fails
+                // for every previously-applied TX, which:
+                //   * pollutes logs with [REJECT][TX] invalid_nonce noise
+                //   * causes block-level apply failure if any TX inside fails
+                //   * cascades to state divergence between nodes that received the
+                //     block once vs nodes that re-applied it multiple times
+                //
+                // ROOT CAUSE OF NETWORK HALT (observed at h=350):
+                //   Genesis block re-delivered during catch-up sync. Sender "genesis"
+                //   had nonce=100 from initial apply. Re-apply attempted nonces 1..100
+                //   sequentially, all rejected. Block partially applied → state_root
+                //   diverged from peers → next block (h=351) failed hash_chain_break →
+                //   pipeline jammed forever.
+                //
+                // SAFETY: silent skip is NOT a security relaxation:
+                //   * If `tx.nonce <= sender.nonce`, the operation has already taken
+                //     effect on this account. Re-applying would either no-op
+                //     (idempotent) or fail (current behaviour) — both leave state
+                //     identical. Silent skip preserves the same final state without
+                //     polluting the failure path.
+                //   * Replay-attack semantics are preserved: an attacker re-broadcasting
+                //     a signed TX with old nonce cannot double-spend, because the
+                //     sender's balance already reflects the original deduction. The
+                //     skipped TX has no incremental effect.
+                //   * Strict +1 check still applies for FUTURE nonces; only stale
+                //     (≤ current) nonces are silently skipped.
+                //
+                // SCALABILITY: O(1) per TX — single comparison. Identical cost at
+                // 5 or 5000 validators. Scales to thousands of super-nodes without
+                // any cross-node coordination.
+                // ═══════════════════════════════════════════════════════════════
+                if self.nonce <= sender.nonce {
+                    // Already applied — silent no-op. Preserves idempotency under
+                    // replay/re-sync without state divergence.
+                    return Ok(());
+                }
+
                 // CRITICAL SECURITY: Check nonce to prevent replay attacks and double spending
                 // Transaction nonce must be exactly sender.nonce + 1
                 if self.nonce != sender.nonce + 1 {
@@ -1576,8 +1619,15 @@ impl Transaction {
                     .ok_or_else(|| StateError::InvalidTransaction("[REJECT][TRANSFER] receiver_balance_overflow".into()))?;
             }
             TransactionType::CreateAccount { address, initial_balance } => {
+                // IDEMPOTENT APPLY — re-creation is a no-op, not an error.
+                // When genesis-style blocks are re-delivered during sync, every
+                // CreateAccount in that block is re-presented. Returning Err here
+                // would fail the whole block apply and corrupt subsequent state;
+                // returning Ok preserves idempotency without changing semantics
+                // (the account already exists with its initial balance, mint cannot
+                // be repeated because the contains_key short-circuit prevents it).
                 if accounts.contains_key(address) {
-                    return Err(StateError::InvalidTransaction("[REJECT][CREATE-ACCOUNT] account_already_exists".to_string()));
+                    return Ok(());
                 }
 
                 // C1 SECURITY: Only system/genesis accounts can mint initial balance
@@ -1621,21 +1671,24 @@ impl Transaction {
                 let sender = accounts.get_mut(&self.from)
                     .expect("account just inserted");
 
-                // v14.8.4: SINGLE-USE ACTIVATION GUARD.
+                // v14.8.4: SINGLE-USE ACTIVATION GUARD (with idempotent re-apply).
                 // Each wallet may hold exactly one active node at a time. An
-                // already-activated wallet trying to submit another NodeActivation
-                // is a replay (same wallet, different burn_tx/activation_code) or a
-                // misconfigured relaunch. Reject at state apply so it cannot be
-                // re-applied from the mempool or from gossip.
+                // already-activated wallet re-presented during batch sync (same
+                // block re-delivered) must be a no-op, not an error — re-apply
+                // would otherwise fail the whole block and corrupt subsequent
+                // state. The mempool layer prevents fresh NodeActivation TXs
+                // from already-activated wallets; this code path only fires on
+                // sync replay where idempotency is the correct semantic.
                 if sender.is_node {
-                    return Err(StateError::InvalidTransaction(format!(
-                        "[REJECT][TX] wallet_already_activated from={} existing_type={:?}",
-                        self.from, sender.node_type
-                    )));
+                    return Ok(());
                 }
 
                 // CRITICAL SECURITY: Check nonce to prevent replay attacks.
                 // First-time wallet has sender.nonce == 0 → valid TX nonce is 1.
+                // Idempotent skip for already-applied: tx.nonce ≤ sender.nonce.
+                if self.nonce <= sender.nonce {
+                    return Ok(());
+                }
                 if self.nonce != sender.nonce + 1 {
                     return Err(StateError::InvalidTransaction(format!(
                         "[REJECT][TX] invalid_nonce expected={} got={}",
@@ -1680,7 +1733,12 @@ impl Transaction {
                 // which is part of the Merkle tree -> replicated to ALL nodes via blocks
                 let sender = accounts.get_mut(&self.from)
                     .ok_or_else(|| StateError::AccountNotFound(self.from.clone()))?;
-                
+
+                // IDEMPOTENT APPLY — see Transfer arm for full rationale. Re-presented
+                // ContractDeploy with stale nonce is a no-op (already deployed).
+                if self.nonce <= sender.nonce {
+                    return Ok(());
+                }
                 // CRITICAL SECURITY: Check nonce to prevent replay attacks
                 if self.nonce != sender.nonce + 1 {
                     return Err(StateError::InvalidTransaction(format!(
@@ -1790,7 +1848,11 @@ impl Transaction {
                 // transfer, approve, transferFrom all modify contract_storage in blockchain state
                 let sender = accounts.get_mut(&self.from)
                     .ok_or_else(|| StateError::AccountNotFound(self.from.clone()))?;
-                
+
+                // IDEMPOTENT APPLY — see Transfer arm for full rationale.
+                if self.nonce <= sender.nonce {
+                    return Ok(());
+                }
                 // CRITICAL SECURITY: Check nonce to prevent replay attacks
                 if self.nonce != sender.nonce + 1 {
                     return Err(StateError::InvalidTransaction(format!(
@@ -2051,7 +2113,11 @@ impl Transaction {
                 // v3.18: Gas fee goes directly to block producer (Pool 2 removed)
                 let sender = accounts.get_mut(from)
                     .ok_or_else(|| StateError::AccountNotFound(from.clone()))?;
-                
+
+                // IDEMPOTENT APPLY — see Transfer arm for full rationale.
+                if self.nonce <= sender.nonce {
+                    return Ok(());
+                }
                 // CRITICAL SECURITY: Check nonce to prevent replay attacks
                 if self.nonce != sender.nonce + 1 {
                     return Err(StateError::InvalidTransaction(format!(
@@ -2194,6 +2260,10 @@ impl Transaction {
                 let sender = accounts.get_mut(&self.from)
                     .ok_or_else(|| StateError::AccountNotFound(self.from.clone()))?;
 
+                // IDEMPOTENT APPLY — see Transfer arm for full rationale.
+                if self.nonce <= sender.nonce {
+                    return Ok(());
+                }
                 // CRITICAL SECURITY: Check nonce to prevent replay attacks
                 if self.nonce != sender.nonce + 1 {
                     return Err(StateError::InvalidTransaction(format!(
@@ -2231,6 +2301,10 @@ impl Transaction {
                 let sender = accounts.get_mut(&self.from)
                     .ok_or_else(|| StateError::AccountNotFound(self.from.clone()))?;
 
+                // IDEMPOTENT APPLY — see Transfer arm for full rationale.
+                if self.nonce <= sender.nonce {
+                    return Ok(());
+                }
                 // CRITICAL SECURITY: Check nonce to prevent replay attacks
                 if self.nonce != sender.nonce + 1 {
                     return Err(StateError::InvalidTransaction(format!(
@@ -2280,6 +2354,10 @@ impl Transaction {
                 let sender = accounts.get_mut(&self.from)
                     .ok_or_else(|| StateError::AccountNotFound(self.from.clone()))?;
 
+                // IDEMPOTENT APPLY — see Transfer arm for full rationale.
+                if self.nonce <= sender.nonce {
+                    return Ok(());
+                }
                 // CRITICAL SECURITY: Check nonce to prevent replay attacks
                 if self.nonce != sender.nonce + 1 {
                     return Err(StateError::InvalidTransaction(format!(
@@ -2493,6 +2571,10 @@ impl Transaction {
                 let account = accounts.get_mut(&self.from)
                     .expect("account just inserted");
 
+                // IDEMPOTENT APPLY for PQ-upgrade — see Transfer arm for rationale.
+                if self.nonce <= account.nonce {
+                    return Ok(());
+                }
                 // Nonce monotonicity (same as any user TX).
                 if self.nonce != account.nonce + 1 {
                     return Err(StateError::InvalidTransaction(format!(