dcrawl - Fast Ethereum Crawler

Introduction

Tool to crawl the Ethereum network, based on the Nim implementation of the Discovery v5 protocol, part of the nim-eth library.

Prerequisites

To build and run, thanks to the included nimbus-build-system, the tool is almost self-contained — you only need git and a C compiler.

For postprocessing (make plots), you also need uv — both postprocessing/plot.py and postprocessing/progress.py use PEP 723 inline metadata so uv run resolves their Python dependencies automatically.

Quick start

# download dependencies and build compiler
make -j4 update
# build the crawler
make -j4
# run the crawler against Ethereum bootnodes
./run.sh
# render charts for the latest run
make plots

./run.sh creates results/<ISO-timestamp>/ and cds into it before launching, so all output (peerstore.csv, discovery.csv, agents.csv, the chronicles JSON log dcrawl.log) lives under that directory. make plots picks the most recent results/*/ and renders charts there.

Configuration options:

./run.sh --help

Client identification

Three identification approaches are available, increasing in cost and accuracy. The first two (--guess, --identify) are runtime modes; the third is an offline classifier metrics summary from postprocessing/fingerprint.py against labeled samples.

--guess (cheap, ENR-only)

Adds a guess column to peerstore.csv and discovery.csv derived from the ENR alone — no extra network traffic. Rules are hand-distilled from a decision tree trained in postprocessing/fingerprint.py against labeled samples; see that script and rerun it on a fresh capture if you want to refresh the rule set.

What's reliably identifiable from ENR alone:

Stack	Strong signal	Partial / ambiguous	Falls back to
CL	teku (no quic), Prysm (high enr_seq), Lighthouse-Pulse / Prysm-Pulse (no nfd/cgc), hermes (cgc only + high enr_seq)	lodestar / Grandine / tysm collapse into Lighthouse — same ENR shape	Lighthouse|lodestar|Grandine|rust-libp2p
EL	erigon (wit key)	bor / BeraGeth / atlas / Gmeme indistinguishable from Geth via ENR alone; ~10% of erigon ENRs lack the wit key and also fall through here	Geth|bor|BeraGeth|erigon
op-stack	opstack ENR key	optimism / base / taiko / kona / mantle / conduit / worldchain all share the same opstack shape — emitted as a multi-label listing all known sub-clients	n/a
Other	ssv ENR key → SSV-Node		n/a

When --guess runs alongside --identify, the crawler tracks how often each guess matches the actual identified client family. The identify stats log line reports guess_match and guess_mismatch totals every --stats-interval-s; the final summary prints a breakdown of the most frequent mismatch patterns (e.g. op-stack|optimism|...|mantle->kona, Geth|bor|BeraGeth|erigon->reth) — useful for spotting when rules drift as clients release new versions. These periodic events go to dcrawl.log (the JSON sink) rather than stdout, since stdout filters at --log-level (default INFO) while progress events use a separate chronicles stream.

Multi-label guesses. When the ENR pattern is genuinely ambiguous, the guess is a |-separated list of candidates (leftmost = most likely). The match counter credits any candidate matching the identified client as a hit, so the accuracy stat stays honest while the user-facing label admits uncertainty. Current multi-label rules (calibrated against an ~11k-peer mainnet sample with --identify, accuracy ~96%; same run that feeds the n=5093 CL / n=4617 EL labeled subsets in the metrics tables below):

• Lighthouse-Pulse|Lighthouse — Pulse fork ENRs are confused with mainline Lighthouse ~30–50% of the time
• Prysm-Pulse|Prysm — same story for Prysm
• Lighthouse|lodestar|Grandine|rust-libp2p — the dominant nfd+cgc+quic low-seq pattern; Lighthouse dominates but lodestar (~5%), Grandine, and rust-libp2p collapse onto the same ENR shape
• Geth|bor|BeraGeth|erigon — eth-only with no wit defaults to Geth but covers bor / BeraGeth and the ~10% of erigon ENRs that don't advertise wit
• op-stack|optimism|base|taiko|kona|worldchain|conduit|mantle — every OP-Stack client shares the same opstack ENR shape, so the multi-label enumerates all known sub-clients

postprocessing/plot.py renders clients-guess.png as a pie of the verbatim guess column, so multi-label slices show alongside crisp ones — making the rule set's uncertainty visible at a glance.

Limitations:

• The cgc-without-nfd rule that maps to hermes will pick up some unidentified CL peers with stale ENRs as false positives. hermes count in the guess distribution is approximate.
• EL fingerprinting from ENR alone is essentially Geth-vs-erigon-with-wit; the no-wit default bucket collapses Geth, bor, BeraGeth, atlas, Gmeme, reth, Nethermind, Besu, and the ~10% of erigon ENRs that don't advertise wit. The multi-label Geth|bor|BeraGeth|erigon enumerates only the clients we calibrated against (everything else still lands here, just isn't named in the label). Real distinction needs the devp2p Hello.

--identify (libp2p identify + devp2p Hello)

Runs an actual handshake on each peer's TCP port to pull the real clientId. Slower (seconds per peer at the timeout, parallelized via worker pool) but high-confidence when peers respond — note the clientId / agent_version is self-reported by the peer, so this is more "real handshake" than absolute ground truth:

• CL peers get a libp2p identify dial. Captures agent_version, protocol_version, and the supported protocols list. ~14% hit rate on mainnet (most peers are firewalled/NATed).
• EL peers (ENR has eth: but not eth2:) get a devp2p Hello dial via RLPx. Captures clientId plus the Disconnect reason code peers send back when they see our empty capabilities list — that reason itself is a strong fingerprint (Geth → 3, erigon → 4, reth → 6).

Output goes to agents.csv with columns cycle, node_id, ip:port, agent_version, protocol_version, protocols, attempt, stack, disconnect_reason, guess. Retries are organized in cycles (--identify-cycles, --identify-cycle-min-interval-s); see --help.

Identifiability summary on a recent mainnet sample

After both libp2p identify and devp2p Hello (with disconnect-reason capture), per-client classifier metrics from postprocessing/fingerprint.py cross-validation on the 2026-04-30 mainnet run:

CL clients (n=5093 labeled)

Client	Precision	Recall	F1	Verdict
Lighthouse	0.97	0.98	0.98	strong
Prysm	0.99	0.96	0.97	strong (high enr_seq)
teku	0.97	1.00	0.98	strong
hermes	1.00	1.00	1.00	strong (cgc key)
erigon (CL)	0.92	0.79	0.85	strong (wit carries over from EL stack)
rust-libp2p	0.91	0.83	0.87	strong
lodestar	0.91	0.74	0.82	recoverable from key-count + syncnets
Lighthouse-Pulse	0.72	0.86	0.79	partial — confused with Lighthouse
Prysm-Pulse	0.68	0.90	0.77	partial — confused with Prysm
Grandine	0.00	0.00	0.00	structurally indistinguishable from Lighthouse

EL clients (n=4617 labeled, with disconnect_reason)

Client	Precision	Recall	F1	Verdict
Geth	0.92	1.00	0.96	strong
erigon	0.99	0.97	0.98	strong (wit key + reason 4)
reth	1.00	0.93	0.96	strong (reason 6 unique to reth)
BeraGeth	0.90	0.38	0.53	high precision, low recall
bor	0.89	0.14	0.24	high precision but most bor → Geth
Gmeme	0.00	0.00	0.00	indistinguishable from Geth

At a 0.8 confidence threshold, classifier-based identification covers ~86% of CL peers at 99.1% accuracy and ~96% of EL peers at 93.9% accuracy. The remaining tail needs the real probe to resolve.

License

Licensed and distributed under either of

• MIT license: LICENSE-MIT or http://opensource.org/licenses/MIT

or

• Apache License, Version 2.0, (LICENSE-APACHEv2 or http://www.apache.org/licenses/LICENSE-2.0)

at your option. This file may not be copied, modified, or distributed except according to those terms.