Skip to content

Commit 732d93f

Browse files
fjahrfjahr
committed
Add BIP UTXO set sharing
1 parent 805c9b5 commit 732d93f

1 file changed

Lines changed: 228 additions & 0 deletions

File tree

‎bip-XXXX.md‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
```
2+
BIP: ?
3+
Layer: Peer Services
4+
Title: P2P UTXO Set Sharing
5+
Authors: Fabian Jahr <fjahr@protonmail.com>
6+
Status: Draft
7+
Type: Specification
8+
Assigned: ?
9+
Discussion: ?
10+
Version: 0.2.0
11+
License: BSD-2-Clause
12+
```
13+
14+
## Abstract
15+
16+
This BIP defines a P2P protocol extension for sharing full UTXO sets between peers. It introduces
17+
a new service bit `NODE_UTXO_SET`, four new P2P messages (`getutxotree`, `utxotree`, `getutxoset`,
18+
`utxoset`), and a chunk-hash list anchored to a Merkle root known to the requesting node, enabling
19+
per-chunk verification. This allows nodes to bootstrap from a recent height by obtaining the
20+
required UTXO set directly from the P2P network via mechanisms such as assumeutxo.
21+
22+
## Motivation
23+
24+
The assumeutxo feature (implemented in Bitcoin Core) allows nodes to begin operating from a serialized
25+
UTXO set while validating
26+
historical blocks in the background. However, there is currently no canonical source for obtaining this
27+
data. Users must either generate one themselves from a fully synced node (using `dumptxoutset` in
28+
Bitcoin Core), or download one from a third party.
29+
30+
By enabling UTXO set sharing over the P2P network, new nodes can obtain the data directly from
31+
peers, removing the dependency on external infrastructure.
32+
33+
## Specification
34+
35+
The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in this document are to be
36+
interpreted as described in RFC 2119.
37+
38+
### Service Bit
39+
40+
| Name | Bit | Description |
41+
|------|-----|-------------|
42+
| `NODE_UTXO_SET` | 12 (0x1000) | The node can serve complete UTXO set data for at least one height. |
43+
44+
A node MUST NOT set this bit unless it has at least one full UTXO set available to serve.
45+
A node signaling `NODE_UTXO_SET` MUST be capable of responding to `getutxotree` and `getutxoset`
46+
requests for every UTXO set it is willing to serve, including the full chunk-hash list and every
47+
chunk of those sets.
48+
49+
### Data Structures
50+
51+
#### Serialized UTXO Set
52+
53+
The serialized UTXO set uses the format established by the Bitcoin Core RPC `dumptxoutset` (as of Bitcoin Core v31).
54+
55+
**Header (55 bytes):**
56+
57+
| Field | Type | Size | Description |
58+
|-------|------|------|-------------|
59+
| `magic` | `bytes` | 5 | `0x7574786fff` (ASCII `utxo` + `0xff`). |
60+
| `version` | `uint16_t` | 2 | Format version. |
61+
| `network_magic` | `bytes` | 4 | Network message start bytes. |
62+
| `base_height` | `uint32_t` | 4 | Block height of the UTXO set. |
63+
| `base_blockhash` | `uint256` | 32 | Block hash of the UTXO set. |
64+
| `coins_count` | `uint64_t` | 8 | Total number of coins (UTXOs) in the set. |
65+
66+
**Body (coin data):**
67+
68+
Coins are grouped by transaction hash. For each group:
69+
70+
| Field | Type | Size | Description |
71+
|-------|------|------|-------------|
72+
| `txid` | `uint256` | 32 | Transaction hash. |
73+
| `num_coins` | `compact_size` | 1–9 | Number of outputs for this txid. |
74+
75+
For each coin in the group:
76+
77+
| Field | Type | Size | Description |
78+
|-------|------|------|-------------|
79+
| `vout_index` | `compact_size` | 1–9 | Output index. |
80+
| `coin` | `Coin` | variable | Serialized coin (varint-encoded code for height/coinbase, then compressed txout). |
81+
82+
Coins are ordered lexicographically by outpoint (txid, then vout index), matching the LevelDB iteration
83+
order of the coins database.
84+
85+
#### Chunk Merkle Tree
86+
87+
The serialized UTXO set (header + body) is split into chunks of exactly 3,900,000 bytes (3.9 MB). The
88+
last chunk contains the remaining bytes and may be smaller.
89+
90+
The leaf hash for each chunk is `SHA256d(chunk_data)`. The tree is built as a balanced binary tree. When
91+
the number of nodes at a level is odd, the last node is duplicated before hashing the next level.
92+
Interior nodes are computed as `SHA256d(left_child || right_child)`.
93+
94+
The leaves are delivered to the node in a single `utxotree` response. A node that knows
95+
the Merkle root for a given UTXO set checks a received list of leaves by recomputing the root and
96+
comparing. The Merkle root is the sole trust input required to verify the integrity of the received UTXO set.
97+
98+
`SHA256d` denotes double-SHA256: `SHA256d(x) = SHA256(SHA256(x))`.
99+
100+
### Messages
101+
102+
#### `getutxotree`
103+
104+
Sent to request the chunk-hash list for a specific UTXO set.
105+
106+
| Field | Type | Size | Description |
107+
|-------|------|------|-------------|
108+
| `block_hash` | `uint256` | 32 | Block hash identifying the requested UTXO set. |
109+
110+
A node that has advertised `NODE_UTXO_SET` and can serve the requested UTXO set MUST respond with
111+
`utxotree`. If the serving node cannot fulfill the request, it MUST NOT respond. The requesting
112+
node SHOULD apply a reasonable timeout and try another peer.
113+
114+
#### `utxotree`
115+
116+
Sent in response to `getutxotree`, delivering the full chunk-hash list along with per-snapshot
117+
metadata.
118+
119+
| Field | Type | Size | Description |
120+
|-------|------|------|-------------|
121+
| `block_hash` | `uint256` | 32 | Block hash this data corresponds to. |
122+
| `version` | `uint16_t` | 2 | Format version of the serialized UTXO set. |
123+
| `data_length` | `uint64_t` | 8 | Total size of the serialized UTXO set in bytes (header + body). |
124+
| `num_chunks` | `compact_size` | 1–9 | Number of chunks the serialized UTXO set is split into. |
125+
| `chunk_hashes` | `uint256[]` | 32 × `num_chunks` | The ordered list of chunk hashes. |
126+
127+
Upon receiving a `utxotree` message, the node MUST recompute the Merkle root from
128+
`chunk_hashes` and compare it against the Merkle root it knows for the corresponding UTXO set. If
129+
the roots do not match, the node MUST discard the response and MUST disconnect the peer.
130+
131+
#### `getutxoset`
132+
133+
Sent to request a single chunk of UTXO set data. The requesting node MUST have received a `utxotree`
134+
for the corresponding UTXO set before sending this message.
135+
136+
| Field | Type | Size | Description |
137+
|-------|------|------|-------------|
138+
| `block_hash` | `uint256` | 32 | Block hash identifying the requested UTXO set. |
139+
| `chunk_index` | `uint32_t` | 4 | Zero-based index of the requested chunk. |
140+
141+
If the serving node cannot fulfill the request, it MUST NOT respond. The requesting node SHOULD apply
142+
a reasonable timeout and try another peer.
143+
144+
#### `utxoset`
145+
146+
Sent in response to `getutxoset`, delivering one chunk.
147+
148+
| Field | Type | Size | Description |
149+
|-------|------|------|-------------|
150+
| `block_hash` | `uint256` | 32 | Block hash this data corresponds to. |
151+
| `chunk_index` | `uint32_t` | 4 | Zero-based index of this chunk. |
152+
| `data` | `bytes` | variable | Chunk payload, exactly 3.9 MB except for the last chunk. |
153+
154+
The transfer is receiver-driven: the requesting node sends one `getutxoset` per chunk. Chunks MAY be
155+
requested in any order and from different peers.
156+
157+
Upon receiving a `utxoset` message, the node MUST compute `SHA256d(data)` and compare it against
158+
`chunk_hashes[chunk_index]` from the `utxotree` it accepted for this UTXO set. If the hashes do not
159+
match, the node MUST discard the chunk and MUST disconnect the peer. A node SHOULD also disconnect
160+
a peer that sends a `utxoset` message with fields (`chunk_index`, `block_hash`) that do not match
161+
the outstanding request.
162+
163+
After all chunks have been received, the node SHOULD parse the reassembled UTXO set against the
164+
serialized UTXO set format to confirm it is well-formed.
165+
166+
### Protocol Flow
167+
168+
1. The requesting node identifies peers advertising `NODE_UTXO_SET`.
169+
2. The requesting node sends `getutxotree` for the desired block hash to one or more of these peers.
170+
3. Each peer responds with `utxotree`. The requesting node verifies the response by recomputing
171+
the Merkle root against a value it knows for the given UTXO set, either from a trusted source
172+
or by selecting a root with agreement among multiple peers.
173+
4. The requesting node downloads chunks via `getutxoset`/`utxoset` exchanges, verifying each chunk
174+
against its entry in the accepted `utxotree` on receipt. On verification failure the peer is
175+
disconnected and download continues from another peer without losing already-verified chunks.
176+
5. After all chunks are received, the node parses the reassembled UTXO set against the serialized
177+
UTXO set format to confirm that it is well-formed.
178+
179+
Serving nodes are free to limit the number of concurrent and repeated transfers per peer at their own
180+
discretion to manage resource consumption.
181+
182+
## Rationale
183+
184+
**Usage of service bit 12:** Service bits allow selective peer discovery through
185+
DNS seeds and addr relay. Bit 12 is chosen as the next unassigned bit after `NODE_P2P_V2` (bit 11, BIP 324).
186+
187+
**Direct request model:** Peers signal availability of UTXO sets via the `NODE_UTXO_SET`
188+
service bit; the requesting node identifies the desired UTXO set by block hash when sending
189+
`getutxotree`. The serving node responds only if it can serve that specific UTXO set.
190+
191+
**Per-chunk verification:** The chunk-hash list returned in `utxotree` enables each chunk to be verified
192+
by direct lookup against the accepted list as it arrives, allowing immediate detection of corrupt data,
193+
peer switching without data loss, and parallel download from multiple peers. The list itself is small
194+
(~80 KB for a ~10 GB set). The specified serialization is deterministic, so all honest nodes produce
195+
byte-identical output, guaranteeing Merkle root agreement.
196+
197+
**3.9 MB chunk size:** The number balances round trips (~2,560 for a ~10 GB set) against memory usage
198+
for buffering and verifying a single chunk. Smaller chunks would increase protocol overhead; larger
199+
chunks would increase memory pressure on constrained devices commonly used to run Bitcoin nodes.
200+
Together with the additional message overhead, the `utxoset` message including the chunk data also
201+
sits just below the theoretical maximum block size which means any implementation should be able to
202+
handle messages of this size.
203+
204+
**Reusing the `dumptxoutset` format:** Avoids introducing a new serialization format and ensures
205+
compatibility with UTXO sets already being generated and shared.
206+
207+
**Relationship to BIP 64:** BIP 64 defined a protocol for querying individual UTXOs by outpoint and is
208+
now closed. This BIP addresses a different use case: bulk transfer of the entire UTXO set for node
209+
bootstrapping.
210+
211+
## Reference Implementation
212+
213+
[Bitcoin Core implementation pull request](https://github.com/bitcoin/bitcoin/pull/35054)
214+
215+
## Copyright
216+
217+
This BIP is made available under the terms of the 2-clause BSD license. See
218+
https://opensource.org/license/BSD-2-Clause for more information.
219+
220+
## Changelog
221+
222+
* __0.2.0__ (2026-05-04):
223+
* Dropped discovery before download approach, instead request the chunk-hash list via `getutxotree`/`utxotree`
224+
* Dropped per-chunk Merkle proofs; chunks verified directly against the chunk-hash list
225+
* Dropped `height` from requests (`block_hash` is the sole identifier); added format `version` to `utxotree`
226+
* Dropped references to the serialized hash; the Merkle root is the sole integrity check
227+
* __0.1.0__ (2026-04-10):
228+
* Initial draft

0 commit comments

Comments
 (0)