Test snap sync of all EL clients on 10x mainnet bloated state

[marcindsobczak]

Measure snap sync of the x10 milestone — the headline result. Same metric set + setup as #16, at the x10 state.

Sync pairs — server held constant (NM), syncer varied; + one realism run

• NM → NM — NM sync-cost scaling + snap-synced NM DB size
• NM → Geth — Geth sync-cost scaling; produces the bootable Geth x10 snapshot
• Geth → NM — mainnet-realism run: on mainnet NM syncs mostly from Geth peers, so this checks whether the real-world server mix changes NM's sync cost (high-value at the top milestone)
• (opportunistic) Geth → Geth — server-side effects only

Metrics (per pair)

• Wall time / bandwidth / IOPS / RSS / final DB size

Acceptance

• NM→NM, NM→Geth (and Geth→NM) complete VALID at x10
• Metrics committed to benchmarks/x10/ per pair
• Compared vs x7 (flag >25% regressions)

Depends on #12. Boundary: sync only; exec+RPC in #21.

[AnkushinDaniil]

NM → NM snap sync — x10 ✅ completed

Cross-VM snap sync of the x10 milestone (10.03× mainnet, ~2.22 TB state). Server held constant (frozen NM x10), syncer = fresh NM.

Result

• Status: Snap sync completed / STATE SYNC FINISHED — converged to the pivot state root, i.e. bit-identical to the source.
• Pivot: block 24,759,753 (head−16), state root 0x2c58426e78b133f62ba86ba09f95f0c116e9acc0e15f74405afd7f674651e94a (source's root at that block). Bit-identical is proven by heal-convergence — NM only reports completion once the local trie matches the pivot root. (Frozen source has no advancing CL, so the synced head stays at genesis and isn't directly queryable via eth_getBlockByNumber("latest"); same limitation as the x1/x3.5 runs.)

Metrics (NM → NM)

metric	value
wall time	27 h 24 m (25 Jun 08:14:10 → 26 Jun 11:37:49 UTC)
final DB size (target)	3.16 TB
avg state throughput	~115 GB/h (~32 MB/s) — single-peer-limited
peers	1 (the single frozen source)
peak RSS	~35 GiB (host 62 GiB; no OOM)
client	nethermind:wip-v22-staticpivot (v1.39.0), both sides

Headline finding — the storage-spam mega-trie tail dominates

Account ranges + ordinary storage finished comparatively early; the final two "large storage" tries took the entire back half of the run, and the last single storage trie alone took ~9 h (~⅓ of total wall), streaming at a flat ~6 %/h with the % barely moving. This is the x10 manifestation of the storage-spam clones (≈3.31 M contracts share an identical storage root) — one giant trie serialises the snap tail. Same shape as the x3.5 cross-client run, amplified at x10.

Setup notes / gotchas (cross-VM NM→NM on this bloatnet)

• Both nodes wip-v22-staticpivot, host networking + Init.DiscoveryEnabled=true — Sync.NetworkingEnabled=true alone does not start the RLPx server (no listener, 0 peers).
• Network.FilterPeersByRecentIp=false + Network.FilterPeersBySameSubnet=false on both — otherwise the source throttles the target IP after a few failed handshakes (state persists in peers/SimpleFileDb.db) → 0 peers on every restart after the first.
• No CL on the isolated net → drive snap with a minimal fake-CL FCU loop: engine_forkchoiceUpdatedV3(head=24759769, safe=finalized=24759753) every 12 s. NM takes the finalized block as the snap pivot.
• Frozen source serves the pivot from its flat DB within Sync.SnapServingMaxDepth=128 of head — pivot at head−16 is well inside the window.

Topology: source = VM3 (91.134.37.235), target = VM2. Source restored to its frozen/offline config afterwards.

[AnkushinDaniil]

NM → Geth snap sync — x10 ✅ completed

Cross-node snap sync of the x10 milestone (10.03× mainnet, ~2.22 TB source state), server held constant (frozen NM x10), syncer = fresh Geth. This is the run that produces the bootable Geth x10 snapshot (deliverable in #32).

Result — VERIFIED against the canonical root

• Status: state sync complete; trie healed.
• Head: block 24,759,769 (0x179cdd9), hash 0xe6aea961…38a2bd.
• State root: 0x5560e21c5ad0231c24626b1bea4a0d591011dbea379e71364d632b3624d571f0 — exact match to the canonical x10 root. Unlike the frozen-source NM→NM run, Geth commits and exposes the head, so this is confirmed directly via eth_getBlockByNumber("latest").stateRoot == canonical (not only by heal-convergence). Bit-exact, bootable Geth x10.
• Healing: 173,204 trie nodes healed (51.6 MB); healingTrienodes=0, stateIndexRemaining=0.

Snapped state (Geth snap counters at completion)

item	count	bytes
accounts	3,944,531,342 (~3.94 B)	834.9 GB
storage slots	14,177,102,805 (~14.18 B)	3.50 TB
bytecodes	712,495,142 (~712 M)	157.8 GB
total snap payload (wire)		~4.49 TB

Metrics (NM → Geth)

metric	value
state-sync wall time	46 h 25 m (05 Jul 04:58:41 → 07 Jul 03:23:11 UTC)
avg state throughput	~26.9 MB/s (~96.8 GB/h) — single-peer-limited
peers	1 (the single frozen NM source)
final DB size (Geth)	~4.1 TiB on-disk (trie + flat snapshot)
NM (server)	nethermind master + flat-serve + repair
Geth (syncer)	v1.17.5, --snap.exactpivot --snap.skipreceipts

(~2 h of prep before the sync: a fresh re-download + extract of the source snapshot — aria2c 1 h 08 m for 3.55 TiB, extract 48 m — after a corrupt local copy was discarded.)

Headline finding — same storage-spam mega-trie tail, serialised behind one peer

The download was lumpy: account ranges sprint (~5–6 %/h), then freeze for tens of minutes while a single storage-spam mega-trie's slots stream in, then resume — several such grinds fill the tail. Same shape as the NM→NM x10 run and the x3.5 cross-client run, amplified at x10: the ≈3.31 M contracts sharing one storage root serialise the snap tail. Byte-% is misleading past ~90 % (Geth's on-disk snap ~4.1 TiB exceeds NM's flat-DB byte count) — accounts-downloaded/total is the honest progress gauge.

Setup notes / gotchas — three things needed fixing vs x7

1. Geth fork-ID reject. The bloatnet genesis carries mainnet's genesis hash, so Geth applies mainnet's Osaka/BPO forks by wall-clock and rejects NM's Prague-only fork ID as stale (x7 synced earlier, before those forks went active). Fix (Geth-side, no genesis change): --override.osaka 9999999999 --override.bpo1 9999999999 --override.bpo2 9999999999 → Geth's fork ID stops at Prague and the handshake passes.
2. NM block-body RLP limit. x10 blocks have gasLimit 8,000,000,000 (~380 k txs); NM's tx-decode cap = Blocks.MaxGasLimit / 21000 (default 1 G → 47,619 ≪ 380,952), so NM threw RlpLimitException decoding its own block bodies on GetBlockBodies and dropped the peer. Fix (config): serve with --Blocks.MaxGasLimit=9000000000.
3. Frozen-source serving. A properly-flushed source is required to snap-serve the pivot; the flat-serve + canonical-index-repair path is feat(jsonrpc): add debug_repairMainChainToHead nethermind#12257 (debug_repairMainChainToHead) and #12263 (flat-serve on a static pivot). --snap.exactpivot (Geth patch, go-ethereum#35265; NM mirror #11943) pins the pivot at the exact head instead of head−64.

Boundary: sync only (exec + RPC in #21). NM→NM x10 is the comment above; Geth→NM (mainnet-realism) still open. Geth x10 snapshot upload tracked in #32.