Skip to content

Commit 3f18743

Browse files
AIQnetLabclaude
andcommitted
fix: v30 — sync cache attestation + cost-asymmetric DoS shield + identity-bound HealthPing
ROOT CAUSES (post-v29 forensic on live testnet) 1) Empty-batch sync feedback loop: handle_blocks_batch attributed the requested to_height to the sender's last_block_height even when zero blocks were returned. Fresh genesis cluster locked into permanent SYNC mode at a phantom 1600 ceiling, never producing the first block. 2) Cost-asymmetric DoS: an external impersonator could spend ~100 bytes/req firing forged dilithium handshake_proof at QUIC :10876 while the receiver paid TLS state + ~3.3 KB Dilithium parse + ML-DSA-65 verify per attempt. 3) HealthPing identity-squat: verify_health_ping_signature consumed the PK embedded in the message body. Any valid ML-DSA-65 keypair could sign QNET_HEALTH_PING_V1:genesis_node_001:<ts>:<h>, attach its own PK, and have a poisoned height accepted as authentic — the last identity-squat hole at the height-gossip layer. 4) /api/v1/peers was the single most-attractive enumeration / DoS target with no rate-limit, exhausting the warp accept queue under flood. FIXES (all scale to hundreds of thousands of super-node identities) A1 handle_blocks_batch attests max(actual block heights) only; empty batches refresh liveness without raising the cached height. A2 get_max_peer_height + sync_blockchain_height require ≥ 2 attested peers before reporting a non-local network height; single-peer claims no longer steer sync. Initial sync target derives from this quorum-attested view. A3 PeerInfo.last_height_attested_at TTL window (120 s). Height entries are accepted into network_height only when their attestation is fresh; stale or gossip-relayed (unauthenticated) values are excluded. B1 Early IP-identity gate at handle_server_handshake — claimed genesis_node_NNN must originate from its pinned IPv4; registered super- node identity must match its on-chain NodeRegistration endpoint. Mismatch drops the connection before any Dilithium math is run. NODE_ENDPOINT_REGISTRY is populated by chain-applied NodeRegistration / NodeReactivation TX. B2 Per-source-IP failed-handshake token bucket (20 fails / 60 s → 600 s cooldown). On the next attempt the IP is refused pre-TLS via incoming.refuse(). Genesis IPs are never banned. Sharded DashMap, O(1). B3 /api/v1/peers gated through the existing check_api_rate_limit middleware. C1 verify_health_ping_signature now resolves the verifying PK from CONSENSUS_PK_REGISTRY against `from` — message-supplied PK is ignored. Unknown identities are rejected outright. C2 Audit of remaining identity-binding sites: VrfKeyAnnounce, BlockAttestation, EmptySlotAttestation, TimeoutVote already consult the registry. Mobile wallet/RPC paths are wallet-ownership proofs (separate domain) — left intact. Scalability invariants preserved: every new lookup is O(1) DashMap (sharded, lock-free); registry caps mirror existing CONSENSUS_PK_REGISTRY (100 K). No hot-path scanning, no global locks, no protocol-message format changes (wire compat with pre-v30 peers retained). Validation: cargo check + lib tests, 147/147 pass, 0 regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent fbeea2c commit 3f18743

5 files changed

Lines changed: 452 additions & 66 deletions

File tree

development/qnet-integration/src/genesis_constants.rs

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,73 @@ pub fn get_all_vrf_keys() -> HashMap<String, Vec<u8>> {
234234
VRF_PK_REGISTRY.read().clone()
235235
}
236236

237+
// =========================================================================
238+
// v30.B1: NODE ENDPOINT REGISTRY — node_id → canonical IPv4 string.
239+
//
240+
// Closes the cost-asymmetric DoS where an attacker fires forged
241+
// handshake_proof at a victim's QUIC accept loop from any IP, forcing the
242+
// victim to pay TLS state + ~3.3 KB Dilithium parse per attempt. The early
243+
// IP-identity gate (in quic_transport.rs::handle_server_handshake) consults
244+
// this registry to refuse impersonation BEFORE the expensive verify step.
245+
//
246+
// Source of truth: signed NodeRegistration TX in chain state. Populated by
247+
// the block-apply path (cache_node_registrations_from_transactions_with_dashmap)
248+
// for super-node identities. Genesis identities resolve via the pinned
249+
// GENESIS_NODE_IPS table and do not go through this registry.
250+
//
251+
// Scalability: DashMap (sharded, lock-free O(1)). Sized for hundreds of
252+
// thousands of super-node identities; ~24 bytes overhead per entry.
253+
// =========================================================================
254+
255+
use dashmap::DashMap;
256+
257+
/// Capacity ceiling matches CONSENSUS_PK_REGISTRY (100K). Beyond this, eviction
258+
/// uses the LAST_ACTIVITY tracker that already governs PK registry eviction.
259+
const MAX_NODE_ENDPOINT_REGISTRY_SIZE: usize = 100_000;
260+
261+
lazy_static::lazy_static! {
262+
pub static ref NODE_ENDPOINT_REGISTRY: DashMap<String, String> = DashMap::new();
263+
}
264+
265+
/// Strip an "ip:port" or "ip" endpoint to its IPv4/IPv6 part only.
266+
fn endpoint_ip_only(api_endpoint: &str) -> String {
267+
// Accept "scheme://host:port" form too — strip scheme and trailing path.
268+
let after_scheme = api_endpoint.split("://").nth(1).unwrap_or(api_endpoint);
269+
let host_only = after_scheme.split('/').next().unwrap_or(after_scheme);
270+
// IPv6 may use [::1]:8001 form; bracket-strip and split last colon.
271+
if let Some(rest) = host_only.strip_prefix('[') {
272+
if let Some(end) = rest.find(']') {
273+
return rest[..end].to_string();
274+
}
275+
}
276+
host_only.split(':').next().unwrap_or(host_only).to_string()
277+
}
278+
279+
/// Register/refresh a node's canonical endpoint IP. Called on every
280+
/// NodeRegistration / NodeReactivation TX during block apply.
281+
pub fn register_node_endpoint(node_id: &str, api_endpoint: &str) {
282+
let ip = endpoint_ip_only(api_endpoint);
283+
if ip.is_empty() {
284+
return;
285+
}
286+
if NODE_ENDPOINT_REGISTRY.len() >= MAX_NODE_ENDPOINT_REGISTRY_SIZE
287+
&& !NODE_ENDPOINT_REGISTRY.contains_key(node_id)
288+
{
289+
// At capacity — let LAST_ACTIVITY-driven eviction reclaim later.
290+
if std::env::var("QNET_DETAILED_LOGGING").ok().as_deref() == Some("1") {
291+
println!("[WARN][REG] endpoint_registry_full size={}", NODE_ENDPOINT_REGISTRY.len());
292+
}
293+
return;
294+
}
295+
NODE_ENDPOINT_REGISTRY.insert(node_id.to_string(), ip);
296+
}
297+
298+
/// Lookup canonical endpoint IP for `node_id`. Returns None for unbound
299+
/// identities (first-contact / not-yet-registered super-nodes).
300+
pub fn get_node_endpoint_ip(node_id: &str) -> Option<String> {
301+
NODE_ENDPOINT_REGISTRY.get(node_id).map(|e| e.value().clone())
302+
}
303+
237304
// Optional genesis Dilithium anchor loader. If present, the file binds the
238305
// 5 genesis identities to fixed PKs via set_genesis_anchor_pks (immutable
239306
// once installed; any non-matching PK is rejected as a squat). Super-node

development/qnet-integration/src/node.rs

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4678,11 +4678,15 @@ impl BlockchainNode {
46784678
node_id.clone(),
46794679
(node_type.clone(), wallet_address.clone(), api_endpoint.clone())
46804680
);
4681+
// v30.B1: mirror endpoint IP into global registry for the
4682+
// QUIC accept-path IP-identity gate. Chain-authenticated
4683+
// (TX is signature-validated before this code path).
4684+
crate::genesis_constants::register_node_endpoint(node_id, api_endpoint);
46814685
// Level 2: RocksDB persistent cache (forward + reverse index)
46824686
if let Err(e) = storage.save_node_registration(node_id, type_str, wallet_address, INITIAL_REPUTATION) {
46834687
eprintln!("[WARN][REG] cache_from_block_fail node={} err={}", node_id, e);
46844688
} else if is_info() {
4685-
println!("[INFO][REG] cached_from_produced_block node={} wallet={}...",
4689+
println!("[INFO][REG] cached_from_produced_block node={} wallet={}...",
46864690
node_id, &wallet_address[..wallet_address.len().min(16)]);
46874691
}
46884692

development/qnet-integration/src/quic_transport.rs

Lines changed: 224 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -180,11 +180,160 @@ const MAX_CONNECTIONS_PER_IP_UNKNOWN: u32 = 10;
180180

181181
/// v9.3: Check if IP belongs to a genesis node (compile-time known validators).
182182
fn is_genesis_ip(ip: &std::net::IpAddr) -> bool {
183-
let ip_str = ip.to_string();
183+
let ip_str = canonical_ip_str(ip);
184184
crate::genesis_constants::GENESIS_NODE_IPS.iter()
185185
.any(|(genesis_ip, _)| *genesis_ip == ip_str)
186186
}
187187

188+
/// v30.B1: render `IpAddr` as the canonical dotted/colon string used by the
189+
/// pinned tables (IPv4-mapped IPv6 collapsed to IPv4). Without this an
190+
/// IPv4-mapped form (`::ffff:1.2.3.4`) would never match the IPv4 string in
191+
/// `GENESIS_NODE_IPS` and a legitimate IPv4 peer arriving over an IPv6
192+
/// socket would be falsely rejected by the IP-identity gate.
193+
#[inline]
194+
fn canonical_ip_str(ip: &std::net::IpAddr) -> String {
195+
match ip {
196+
std::net::IpAddr::V4(v4) => v4.to_string(),
197+
std::net::IpAddr::V6(v6) => match v6.to_ipv4_mapped() {
198+
Some(v4) => v4.to_string(),
199+
None => v6.to_string(),
200+
},
201+
}
202+
}
203+
204+
// ============================================================================
205+
// v30.B1/B2: cost-asymmetric DoS killer — early IP-identity gate + per-IP
206+
// failed-handshake token bucket.
207+
//
208+
// Attack model: attacker sends ~100-byte UDP datagrams to the QUIC accept
209+
// port; victim pays TLS state + ~3.3 KB Dilithium parse + ML-DSA-65 verify
210+
// per attempt. Without this gate a 5-6 pkt/s flood from a single IP costs
211+
// the receiver tens of thousands of Dilithium verifies per hour. With it
212+
// the attacker is refused before TLS handshake completes; CPU cost on the
213+
// receiver collapses to a single DashMap lookup + counter bump.
214+
//
215+
// Two independent layers:
216+
// B1 (IP-identity gate): once the wire handshake is deserialised, the
217+
// claimed node_id is checked against either the pinned genesis IP
218+
// table OR the registered endpoint registry. Mismatch is conclusive
219+
// impersonation evidence — drop without paying for Dilithium verify.
220+
// B2 (per-IP fail bucket): every failed handshake from a non-genesis IP
221+
// increments a sliding-window counter; on threshold breach the IP is
222+
// refused at accept time for a cooldown period. Genesis IPs are
223+
// NEVER banned (consensus path is privileged).
224+
//
225+
// Scalability: DashMap is sharded → O(1) under contention; ban entries are
226+
// ~40 bytes; even with a million unique attacker IPs total RAM stays under
227+
// 50 MB. Cleanup is implicit: window rollover + cooldown expiry are checked
228+
// inline, so stale entries clear themselves on next touch.
229+
// ============================================================================
230+
231+
const HANDSHAKE_FAIL_WINDOW_SECS: u64 = 60;
232+
const HANDSHAKE_FAIL_THRESHOLD: u64 = 20;
233+
const HANDSHAKE_FAIL_BAN_SECS: u64 = 600;
234+
235+
struct HandshakeFailState {
236+
fail_count: AtomicU64,
237+
window_start_secs: AtomicU64,
238+
banned_until_secs: AtomicU64,
239+
}
240+
241+
impl Default for HandshakeFailState {
242+
fn default() -> Self {
243+
Self {
244+
fail_count: AtomicU64::new(0),
245+
window_start_secs: AtomicU64::new(0),
246+
banned_until_secs: AtomicU64::new(0),
247+
}
248+
}
249+
}
250+
251+
static HANDSHAKE_FAIL_TRACKER: once_cell::sync::Lazy<
252+
DashMap<std::net::IpAddr, HandshakeFailState>
253+
> = once_cell::sync::Lazy::new(DashMap::new);
254+
255+
#[inline]
256+
fn unix_secs_now() -> u64 {
257+
std::time::SystemTime::now()
258+
.duration_since(std::time::UNIX_EPOCH)
259+
.unwrap_or_default()
260+
.as_secs()
261+
}
262+
263+
/// True if `ip` is in active cooldown — caller refuses the connection
264+
/// before any TLS state is allocated. Genesis IPs always return false.
265+
fn is_handshake_ip_banned(ip: std::net::IpAddr) -> bool {
266+
if is_genesis_ip(&ip) {
267+
return false;
268+
}
269+
HANDSHAKE_FAIL_TRACKER
270+
.get(&ip)
271+
.map(|s| s.banned_until_secs.load(Ordering::Relaxed) > unix_secs_now())
272+
.unwrap_or(false)
273+
}
274+
275+
/// Record one failed handshake from `ip`. Window rollover and cooldown
276+
/// promotion happen inline. Genesis IPs are exempt.
277+
fn record_handshake_fail(ip: std::net::IpAddr) {
278+
if is_genesis_ip(&ip) {
279+
return;
280+
}
281+
let now = unix_secs_now();
282+
let entry = HANDSHAKE_FAIL_TRACKER.entry(ip).or_default();
283+
let window_start = entry.window_start_secs.load(Ordering::Relaxed);
284+
if window_start == 0 || now.saturating_sub(window_start) > HANDSHAKE_FAIL_WINDOW_SECS {
285+
entry.window_start_secs.store(now, Ordering::Relaxed);
286+
entry.fail_count.store(1, Ordering::Relaxed);
287+
return;
288+
}
289+
let new_count = entry.fail_count.fetch_add(1, Ordering::Relaxed) + 1;
290+
if new_count >= HANDSHAKE_FAIL_THRESHOLD
291+
&& entry.banned_until_secs.load(Ordering::Relaxed) <= now
292+
{
293+
entry.banned_until_secs.store(now + HANDSHAKE_FAIL_BAN_SECS, Ordering::Relaxed);
294+
if crate::node::is_warn() {
295+
println!(
296+
"[WARN][QUIC] ip_fail_ban ip={} fails={} window={}s cooldown={}s",
297+
ip, new_count, HANDSHAKE_FAIL_WINDOW_SECS, HANDSHAKE_FAIL_BAN_SECS
298+
);
299+
}
300+
}
301+
}
302+
303+
/// Clear fail counter on a confirmed-successful handshake — promotes the
304+
/// IP back to clean state and removes any residual ban.
305+
fn clear_handshake_fail(ip: std::net::IpAddr) {
306+
HANDSHAKE_FAIL_TRACKER.remove(&ip);
307+
}
308+
309+
/// v30.B1: bind claimed node_id to allowed source IP. Returns true if the
310+
/// gate permits the connection to proceed to Dilithium verification.
311+
///
312+
/// * `genesis_node_NNN` MUST originate from its pinned IPv4 in
313+
/// `GENESIS_NODE_IPS` — the 5-entry table is the singular source of
314+
/// truth; any other source is impersonation.
315+
/// * Super-node identity present in `NODE_ENDPOINT_REGISTRY` MUST match
316+
/// the registered endpoint IP — populated by chain-authenticated
317+
/// NodeRegistration TX during block apply (O(1) DashMap).
318+
/// * Unbound identity (no registry record yet) is admitted: this is the
319+
/// first-contact / TOFV window where the peer is about to register via
320+
/// a signed VrfKeyAnnounce or NodeRegistration TX. The cryptographic
321+
/// floor (verify_handshake_proof) still gates everything they assert.
322+
fn ip_identity_gate(claimed_node_id: &str, peer_ip: std::net::IpAddr) -> bool {
323+
let peer_ip_str = canonical_ip_str(&peer_ip);
324+
if claimed_node_id.starts_with("genesis_node_") {
325+
match crate::genesis_constants::genesis_ip_for_node_id(claimed_node_id) {
326+
Some(expected) => expected == peer_ip_str,
327+
None => false,
328+
}
329+
} else {
330+
match crate::genesis_constants::get_node_endpoint_ip(claimed_node_id) {
331+
Some(expected) => expected == peer_ip_str,
332+
None => true,
333+
}
334+
}
335+
}
336+
188337
/// v9.2: TOFU pin lifetime (24 hours). After this, the pin expires and re-pins
189338
/// on next connection. This handles cert rotation on rolling restarts:
190339
/// node restarts with new self-signed cert → peers accept after TTL expires.
@@ -890,6 +1039,19 @@ impl QuicTransport {
8901039

8911040
let peer_addr = incoming.remote_address();
8921041

1042+
// v30.B2: per-source-IP failed-handshake ban — refuse pre-TLS
1043+
// for IPs that crossed the fail threshold within the rolling
1044+
// window. Reclaims the TLS state + Dilithium parse cost an
1045+
// attacker would otherwise extract per packet. Genesis IPs
1046+
// are never banned by design (consensus must stay reachable).
1047+
if is_handshake_ip_banned(peer_addr.ip()) {
1048+
incoming.refuse();
1049+
if crate::node::is_debug() {
1050+
println!("[DBG][QUIC] pre_tls_ip_banned ip={}", peer_addr.ip());
1051+
}
1052+
continue;
1053+
}
1054+
8931055
// FIX R23-P5: Global connection limit check BEFORE TLS handshake.
8941056
// Previously only checked post-handshake (line ~910), wasting TLS CPU.
8951057
const MAX_TOTAL_CONNECTIONS: usize = 500;
@@ -966,6 +1128,7 @@ impl QuicTransport {
9661128
Ok(Ok(connection)) => {
9671129
let hs = Self::handle_server_handshake(
9681130
&connection,
1131+
peer_addr,
9691132
&node_id_clone,
9701133
&cert_serial_clone,
9711134
&node_type_clone
@@ -1131,35 +1294,55 @@ impl QuicTransport {
11311294
/// v2.24: Added timeout to prevent hanging connections
11321295
/// v9.7: Returns (node_id, cert_serial, node_type, block_height) — height enables
11331296
/// immediate BEST_PEER_HEIGHT update instead of waiting 15s for first HealthPing.
1297+
/// v30.B1: `peer_addr` is now part of the signature — the early IP-identity
1298+
/// gate binds the claimed `node_id` to its registered source IP and rejects
1299+
/// impersonation before paying for the Dilithium verify (~3.3 KB parse +
1300+
/// ML-DSA-65 math). Any Err exit increments the per-IP fail counter so
1301+
/// repeat offenders trip the pre-TLS ban at the accept loop.
11341302
async fn handle_server_handshake(
11351303
conn: &Connection,
1304+
peer_addr: SocketAddr,
11361305
our_node_id: &str,
11371306
our_cert_serial: &str,
11381307
our_node_type: &str,
11391308
) -> Result<(String, String, String, u64), String> {
11401309
// v2.24: Timeout for entire handshake (prevents "aborted by peer" errors)
11411310
let handshake_timeout = Duration::from_secs(CONNECT_TIMEOUT_SECS);
1142-
1143-
tokio::time::timeout(handshake_timeout, async {
1311+
1312+
let result = tokio::time::timeout(handshake_timeout, async {
11441313
// Accept bidirectional stream for handshake
11451314
let (mut send, mut recv) = conn.accept_bi().await
11461315
.map_err(|e| format!("Accept stream failed: {}", e))?;
1147-
1316+
11481317
// Receive peer's handshake
11491318
let mut len_buf = [0u8; 4];
11501319
recv.read_exact(&mut len_buf).await.map_err(|e| format!("Read len failed: {}", e))?;
11511320
let len = u32::from_be_bytes(len_buf) as usize;
1152-
1321+
11531322
if len > MAX_MESSAGE_SIZE {
11541323
return Err(format!("Handshake too large: {}", len));
11551324
}
1156-
1325+
11571326
let mut data = vec![0u8; len];
11581327
recv.read_exact(&mut data).await.map_err(|e| format!("Read data failed: {}", e))?;
1159-
1328+
11601329
// v9.7: Backward-compatible deserialization (supports pre-v9.7 nodes)
11611330
let peer_handshake = deserialize_handshake(&data)?;
11621331

1332+
// v30.B1: early IP-identity gate. Reject impersonation BEFORE
1333+
// the ~ms Dilithium verify pass. Genesis identity from a non-
1334+
// pinned IP and registered super-node identity from a different
1335+
// IP than its on-chain endpoint are conclusively rejected.
1336+
if !ip_identity_gate(&peer_handshake.node_id, peer_addr.ip()) {
1337+
if crate::node::is_warn() {
1338+
println!(
1339+
"[WARN][HANDSHAKE] ip_identity_gate_reject side=server node={} src_ip={} action=close",
1340+
peer_handshake.node_id, peer_addr.ip()
1341+
);
1342+
}
1343+
return Err("ip_identity_gate_reject".to_string());
1344+
}
1345+
11631346
// v19: Verify peer's Dilithium identity proof BEFORE sending ours.
11641347
// On Err the connection is aborted — we never reveal our own proof
11651348
// to a peer that supplied a bogus one. On Ok(false) we proceed
@@ -1175,6 +1358,20 @@ impl QuicTransport {
11751358
println!("[INFO][HANDSHAKE] dilithium_proof_verified side=server node={} h={}",
11761359
peer_handshake.node_id, peer_handshake.block_height);
11771360
}
1361+
// v30.A3: Dilithium-verified handshake binds (node_id, block_height)
1362+
// as an authenticated tuple — attest peer height immediately so the
1363+
// sync state machine sees real network state without waiting for the
1364+
// first HealthPing tick. Non-zero heights only; h=0 leaves the
1365+
// attestation unset so a fresh-cluster cold start does not falsely
1366+
// declare consensus on "everyone at 0".
1367+
if peer_handshake.block_height > 0 {
1368+
if let Some(p2p) = crate::node::try_get_p2p() {
1369+
p2p.update_peer_last_seen_with_height(
1370+
&peer_handshake.node_id,
1371+
Some(peer_handshake.block_height),
1372+
);
1373+
}
1374+
}
11781375
}
11791376
Ok(false) => {
11801377
// v19.1: Advisory admit. Three causes possible (in order
@@ -1239,7 +1436,26 @@ impl QuicTransport {
12391436

12401437
// v9.7: Return height from handshake for immediate BEST_PEER_HEIGHT update
12411438
Ok((peer_handshake.node_id, peer_handshake.cert_serial, peer_handshake.node_type, peer_handshake.block_height))
1242-
}).await.map_err(|_| "Handshake timeout".to_string())?
1439+
}).await;
1440+
1441+
// v30.B2: account success/failure for the per-IP fail bucket. A
1442+
// verified handshake clears any residual ban; any error path (gate
1443+
// reject, proof reject, timeout, malformed wire) bumps the counter
1444+
// so a repeat offender trips the pre-TLS ban at the accept loop.
1445+
match result {
1446+
Ok(Ok(outcome)) => {
1447+
clear_handshake_fail(peer_addr.ip());
1448+
Ok(outcome)
1449+
}
1450+
Ok(Err(e)) => {
1451+
record_handshake_fail(peer_addr.ip());
1452+
Err(e)
1453+
}
1454+
Err(_) => {
1455+
record_handshake_fail(peer_addr.ip());
1456+
Err("Handshake timeout".to_string())
1457+
}
1458+
}
12431459
}
12441460

12451461
/// Handle incoming streams from a connection

0 commit comments

Comments
 (0)