Skip to content

Commit ec63487

Browse files
fjahrfjahr
committed
Add BIP UTXO set sharing
1 parent 805c9b5 commit ec63487

1 file changed

Lines changed: 242 additions & 0 deletions

File tree

‎bip-XXXX.md‎

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
```
2+
BIP: ?
3+
Layer: Peer Services
4+
Title: P2P UTXO Set Sharing
5+
Authors: Fabian Jahr <fjahr@protonmail.com>
6+
Status: Draft
7+
Type: Specification
8+
Assigned: ?
9+
License: BSD-2-Clause
10+
Discussion: 2026-05-06: https://groups.google.com/g/bitcoindev/c/rThmyI8ZN3Q
11+
Version: 0.4.0
12+
```
13+
14+
## Abstract
15+
16+
This BIP defines a P2P protocol extension for sharing full UTXO sets between peers. It introduces
17+
a new service bit `NODE_UTXO_SET`, four new P2P messages (`getutxotree`, `utxotree`, `getutxoset`,
18+
`utxoset`), and a chunk-hash list anchored to a Merkle root known to the requesting node, enabling
19+
per-chunk verification. This allows bootstrapping nodes to leapfrog to a recent height by obtaining the
20+
required UTXO set directly from the P2P network via mechanisms such as assumeutxo.
21+
22+
## Motivation
23+
24+
The assumeutxo feature (implemented in Bitcoin Core) allows nodes to begin operating from a serialized
25+
UTXO set while validating
26+
historical blocks in the background. However, there is currently no canonical source for obtaining this
27+
data. Users must either generate one themselves from a fully synced node (using `dumptxoutset` in
28+
Bitcoin Core), or download one from a third party.
29+
30+
By enabling UTXO set sharing over the P2P network, new nodes can obtain the data directly from
31+
peers, removing the dependency on external infrastructure.
32+
33+
## Specification
34+
35+
The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in this document are to be
36+
interpreted as described in RFC 2119.
37+
38+
### Service Bit
39+
40+
| Name | Bit | Description |
41+
|------|-----|-------------|
42+
| `NODE_UTXO_SET` | 14 (0x4000) | The node can serve complete UTXO set data for at least one height. |
43+
44+
A node MUST NOT set this bit unless it has at least one full UTXO set available to serve.
45+
A node signaling `NODE_UTXO_SET` MUST be capable of responding to `getutxotree` and `getutxoset`
46+
requests for every UTXO set that it is willing to serve, including the full chunk-hash list and every
47+
chunk of those sets.
48+
49+
### Data Structures
50+
51+
#### Serialized UTXO Set
52+
53+
The serialized UTXO set uses the format established by the Bitcoin Core RPC `dumptxoutset` (as of Bitcoin Core v31).
54+
55+
**Header (55 bytes):**
56+
57+
| Field | Type | Size | Description |
58+
|-------|------|------|-------------|
59+
| `magic` | `bytes` | 5 | `0x7574786fff` (ASCII `utxo` + `0xff`). |
60+
| `version` | `uint16_t` | 2 | Format version. |
61+
| `network_magic` | `bytes` | 4 | Network message start bytes. |
62+
| `base_height` | `uint32_t` | 4 | Block height of the UTXO set. |
63+
| `base_blockhash` | `uint256` | 32 | Block hash of the UTXO set. |
64+
| `coins_count` | `uint64_t` | 8 | Total number of coins (UTXOs) in the set. |
65+
66+
**Body (coin data):**
67+
68+
Coins are grouped by transaction hash. For each group:
69+
70+
| Field | Type | Size | Description |
71+
|-------|------|------|-------------|
72+
| `txid` | `uint256` | 32 | Transaction hash. |
73+
| `num_coins` | `compact_size` | 1–9 | Number of outputs for this txid. |
74+
75+
For each coin in the group:
76+
77+
| Field | Type | Size | Description |
78+
|-------|------|------|-------------|
79+
| `vout_index` | `compact_size` | 1–9 | Output index. |
80+
| `coin` | `Coin` | variable | Serialized coin (varint-encoded code for height/coinbase, then compressed txout). |
81+
82+
Coins are ordered lexicographically by outpoint (txid, then vout index), matching the LevelDB iteration
83+
order of the coins database.
84+
85+
#### Chunk Merkle Tree
86+
87+
The serialized UTXO set (header + body) is split into chunks of exactly 3,900,000 bytes (3.9 MB). The
88+
last chunk contains the remaining bytes and may be smaller. The chunks form the leaves of a binary
89+
Merkle tree whose root commits to the entire UTXO set.
90+
91+
The leaf hash for each chunk is `SHA256d(chunk_data)`. The tree is built as a balanced binary tree. When
92+
the number of nodes at a level is odd, the last node is promoted unchanged to the next level.
93+
Interior nodes are computed as `SHA256d(left_child || right_child)`.
94+
95+
The leaves are delivered to the node in a single `utxotree` response. A node that knows
96+
the Merkle root for a given UTXO set checks a received list of leaves by recomputing the root and
97+
comparing. The Merkle root is the sole trust input required to verify the integrity of the received UTXO set.
98+
99+
`SHA256d` denotes double-SHA256: `SHA256d(x) = SHA256(SHA256(x))`.
100+
101+
### Messages
102+
103+
#### `getutxotree`
104+
105+
Sent to request the chunk-hash list for a specific UTXO set.
106+
107+
| Field | Type | Size | Description |
108+
|-------|------|------|-------------|
109+
| `block_hash` | `uint256` | 32 | Block hash identifying the requested UTXO set. |
110+
111+
A node that has advertised `NODE_UTXO_SET` and can serve the requested UTXO set MUST respond with
112+
`utxotree`. If the serving node cannot fulfill the request, it MUST NOT respond. The requesting
113+
node SHOULD apply a reasonable timeout and try another peer.
114+
115+
#### `utxotree`
116+
117+
Sent in response to `getutxotree`, delivering the full chunk-hash list along with per-snapshot
118+
metadata.
119+
120+
| Field | Type | Size | Description |
121+
|-------|------|------|-------------|
122+
| `block_hash` | `uint256` | 32 | Block hash this data corresponds to. |
123+
| `version` | `uint16_t` | 2 | Format version of the serialized UTXO set. |
124+
| `data_length` | `uint64_t` | 8 | Total size of the serialized UTXO set in bytes (header + body). |
125+
| `chunk_hashes` | `uint256[]` | 32 × N | The ordered list of N chunk hashes, where N = `ceil(data_length / 3,900,000)`. |
126+
127+
Upon receiving a `utxotree` message, the requesting node MUST recompute the Merkle root from
128+
`chunk_hashes` and compare it against the Merkle root it knows for the corresponding UTXO set. If
129+
the roots do not match, the node MUST discard the response and MUST disconnect the peer.
130+
131+
#### `getutxoset`
132+
133+
Sent to request a single chunk of UTXO set data. The requesting node MUST have received a `utxotree`
134+
for the corresponding UTXO set (from any peer) before sending this message.
135+
136+
| Field | Type | Size | Description |
137+
|-------|------|------|-------------|
138+
| `block_hash` | `uint256` | 32 | Block hash identifying the requested UTXO set. |
139+
| `chunk_index` | `uint32_t` | 4 | Zero-based index of the requested chunk. |
140+
141+
If the serving node cannot fulfill the request, it MUST NOT respond. The requesting node SHOULD apply
142+
a reasonable timeout and try another peer.
143+
144+
#### `utxoset`
145+
146+
Sent in response to `getutxoset`, delivering one chunk.
147+
148+
| Field | Type | Size | Description |
149+
|-------|------|------|-------------|
150+
| `block_hash` | `uint256` | 32 | Block hash this data corresponds to. |
151+
| `chunk_index` | `uint32_t` | 4 | Zero-based index of this chunk. |
152+
| `data` | `bytes` | variable | Chunk payload, exactly 3.9 MB except for the last chunk. |
153+
154+
The transfer is receiver-driven: the requesting node sends one `getutxoset` per chunk. Chunks MAY be
155+
requested in any order and from different peers.
156+
157+
Upon receiving a `utxoset` message, the node MUST compute `SHA256d(data)` and compare it against
158+
`chunk_hashes[chunk_index]` from the `utxotree` it accepted for this UTXO set. If the hashes do not
159+
match, the node MUST discard the chunk and MUST disconnect the peer. A node SHOULD also disconnect
160+
a peer that sends a `utxoset` message with fields (`chunk_index`, `block_hash`) that do not match
161+
the outstanding request.
162+
163+
After all chunks have been received, the node SHOULD parse the reassembled UTXO set against the
164+
serialized UTXO set format to confirm it is well-formed.
165+
166+
### Protocol Flow
167+
168+
1. The requesting node identifies peers advertising `NODE_UTXO_SET`.
169+
2. The requesting node sends `getutxotree` for the desired block hash to one of these peers, or to
170+
several peers to corroborate the Merkle root by agreement if no trusted root is available.
171+
3. The peer or peers respond with `utxotree`. The requesting node verifies each response by
172+
recomputing the Merkle root and comparing it against a value it knows for the given UTXO set,
173+
either from a trusted source or from agreement among multiple peers. A single accepted `utxotree`
174+
can be used as the basis for all subsequent chunk requests for this UTXO set, regardless of
175+
which peer those chunks are fetched from.
176+
4. The requesting node downloads chunks via `getutxoset`/`utxoset` exchanges, verifying each chunk
177+
against its entry in the accepted `utxotree` on receipt. On verification failure the peer is
178+
disconnected and download continues from another peer without losing already-verified chunks.
179+
5. After all chunks are received, the node parses the reassembled UTXO set against the serialized
180+
UTXO set format to confirm that it is well-formed.
181+
182+
Serving nodes are free to limit the number of concurrent and repeated transfers per peer at their own
183+
discretion to manage resource consumption.
184+
185+
## Rationale
186+
187+
**Usage of service bit 14:** Service bits allow selective peer discovery through
188+
DNS seeds and addr relay. Bit 14 is chosen because bits 12 and 13 are reserved by the
189+
Utreexo proposal (BIP 183 draft).
190+
191+
**Direct request model:** Peers signal availability of UTXO sets via the `NODE_UTXO_SET`
192+
service bit; the requesting node identifies the desired UTXO set by block hash when sending
193+
`getutxotree`. The serving node responds only if it can serve that specific UTXO set.
194+
195+
**Per-chunk verification:** The chunk-hash list returned in `utxotree` enables each chunk to be verified
196+
by direct lookup against the accepted list as it arrives, allowing immediate detection of corrupt data,
197+
peer switching without data loss, and parallel download from multiple peers. The list itself is small
198+
(~80 KB for a ~10 GB set). The specified serialization is deterministic, so all honest nodes produce
199+
byte-identical output, guaranteeing Merkle root agreement.
200+
201+
**3.9 MB chunk size:** The number balances round trips (~2,560 for a ~10 GB set) against memory usage
202+
for buffering and verifying a single chunk. Smaller chunks would increase protocol overhead; larger
203+
chunks would increase memory pressure on constrained devices commonly used to run Bitcoin nodes.
204+
Together with the additional message overhead, the `utxoset` message including the chunk data also
205+
sits just below the theoretical maximum block size which means any implementation should be able to
206+
handle messages of this size.
207+
208+
**Reusing the `dumptxoutset` format:** Avoids introducing a new serialization format and ensures
209+
compatibility with UTXO sets already being generated and shared.
210+
211+
**Relationship to BIP 64:** BIP 64 defined a protocol for querying individual UTXOs by outpoint and is
212+
now closed. This BIP addresses a different use case: bulk transfer of the entire UTXO set for node
213+
bootstrapping.
214+
215+
## Backwards Compatibility
216+
217+
This proposal is backward compatible. Peers that do not implement it ignore the new service bit
218+
and never issue the new messages.
219+
220+
## Reference Implementation
221+
222+
[Bitcoin Core implementation pull request](https://github.com/bitcoin/bitcoin/pull/35054)
223+
224+
## Copyright
225+
226+
This BIP is made available under the terms of the 2-clause BSD license. See
227+
https://opensource.org/license/BSD-2-Clause for more information.
228+
229+
## Changelog
230+
231+
* __0.4.0__ (2026-05-18):
232+
* Removed `num_chunks` from `utxotree`
233+
* __0.3.0__ (2026-05-17):
234+
* Moved service bit from 12 to 14 to avoid collision with the Utreexo proposal (BIP 183 draft)
235+
* Changed Merkle tree construction: odd nodes are promoted unchanged rather than duplicated
236+
* __0.2.0__ (2026-05-04):
237+
* Dropped discovery before download approach, instead request the chunk-hash list via `getutxotree`/`utxotree`
238+
* Dropped per-chunk Merkle proofs; chunks verified directly against the chunk-hash list
239+
* Dropped `height` from requests (`block_hash` is the sole identifier); added format `version` to `utxotree`
240+
* Dropped references to the serialized hash; the Merkle root is the sole integrity check
241+
* __0.1.0__ (2026-04-10):
242+
* Initial draft

0 commit comments

Comments
 (0)