Skip to content

Commit 47f2072

Browse files
fjahrfjahr
committed
Add BIP UTXO set sharing
1 parent 805c9b5 commit 47f2072

1 file changed

Lines changed: 282 additions & 0 deletions

File tree

‎bip-XXXX.md‎

Lines changed: 282 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,282 @@
1+
```
2+
BIP: ?
3+
Layer: Peer Services
4+
Title: P2P UTXO Set Sharing
5+
Authors: Fabian Jahr <fjahr@protonmail.com>
6+
Status: Draft
7+
Type: Specification
8+
Assigned: ?
9+
License: BSD-2-Clause
10+
Discussion: 2026-05-06: https://groups.google.com/g/bitcoindev/c/rThmyI8ZN3Q
11+
Version: 0.5.0
12+
```
13+
14+
## Abstract
15+
16+
This BIP defines a P2P protocol extension for sharing full UTXO sets between peers. It introduces
17+
a new service bit `NODE_UTXO_SET`, four new P2P messages (`getutxotree`, `utxotree`, `getutxoset`,
18+
`utxoset`), and a chunk-hash list anchored to a Merkle root known to the requesting node, enabling
19+
per-chunk verification. This allows bootstrapping nodes to leapfrog to a recent height by obtaining the
20+
required UTXO set directly from the P2P network via mechanisms such as assumeutxo.
21+
22+
## Motivation
23+
24+
The assumeutxo feature (implemented in Bitcoin Core) allows nodes to begin operating from a serialized
25+
UTXO set while validating
26+
historical blocks in the background. However, there is currently no canonical source for obtaining this
27+
data. Users must either generate one themselves from a fully synced node (using `dumptxoutset` in
28+
Bitcoin Core), or download one from a third party.
29+
30+
By enabling UTXO set sharing over the P2P network, new nodes can obtain the data directly from
31+
peers, removing the dependency on external infrastructure.
32+
33+
## Specification
34+
35+
The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in this document are to be
36+
interpreted as described in RFC 2119.
37+
38+
### Service Bit
39+
40+
| Name | Bit | Description |
41+
|------|-----|-------------|
42+
| `NODE_UTXO_SET` | 14 (0x4000) | The node serves complete UTXO set data for the scheduled heights (see [Scheduled UTXO Sets](#scheduled-utxo-sets)). |
43+
44+
A node MUST NOT set this bit unless it can serve the UTXO sets at the scheduled heights defined below.
45+
A node signaling `NODE_UTXO_SET` MUST be capable of responding to `getutxotree` and `getutxoset`
46+
requests for the scheduled heights `H1` and `H2`, including the full chunk-hash list and every chunk of
47+
those sets. It MAY additionally serve UTXO sets at other heights.
48+
49+
### Scheduled UTXO Set heights
50+
51+
To make the `NODE_UTXO_SET` service bit meaningful for peer discovery, advertising nodes serve UTXO
52+
sets at a deterministic schedule of block heights derived from the current tip. All heights refer to
53+
blocks on the active most-work chain.
54+
55+
Let:
56+
57+
* `N` = height of the current chain tip
58+
* `M = N - 2016` (the height as of approximately two weeks ago)
59+
* `K = 14112` (7 difficulty adjustment periods, approximately three months)
60+
* `H1 = M - (M mod K)` (the most recent multiple of `K` that is buried by at least 2016 blocks)
61+
* `H2 = H1 - K` (the preceding multiple of `K`)
62+
63+
A node advertising `NODE_UTXO_SET` MUST be able to serve the UTXO sets at heights `H1` and `H2` as
64+
computed from the current tip. Serving two consecutive scheduled heights guarantees an overlap
65+
window: when a new height becomes `H1`, the previous one remains available as `H2`, so an in-progress
66+
download is not interrupted.
67+
68+
The 2016-block offset ensures a height becomes scheduled only once it is buried by approximately two
69+
weeks, which should make it safe from reorganization and gives serving nodes time to produce the
70+
snapshot before it is requested.
71+
72+
### Data Structures
73+
74+
#### Serialized UTXO Set
75+
76+
The serialized UTXO set uses the format established by the Bitcoin Core RPC `dumptxoutset` (as of Bitcoin Core v31).
77+
78+
**Header (55 bytes):**
79+
80+
| Field | Type | Size | Description |
81+
|-------|------|------|-------------|
82+
| `magic` | `bytes` | 5 | `0x7574786fff` (ASCII `utxo` + `0xff`). |
83+
| `version` | `uint16_t` | 2 | Format version. |
84+
| `network_magic` | `bytes` | 4 | Network message start bytes. |
85+
| `base_height` | `uint32_t` | 4 | Block height of the UTXO set. |
86+
| `base_blockhash` | `uint256` | 32 | Block hash of the UTXO set. |
87+
| `coins_count` | `uint64_t` | 8 | Total number of coins (UTXOs) in the set. |
88+
89+
**Body (coin data):**
90+
91+
Coins are grouped by transaction hash. For each group:
92+
93+
| Field | Type | Size | Description |
94+
|-------|------|------|-------------|
95+
| `txid` | `uint256` | 32 | Transaction hash. |
96+
| `num_coins` | `compact_size` | 1–9 | Number of outputs for this txid. |
97+
98+
For each coin in the group:
99+
100+
| Field | Type | Size | Description |
101+
|-------|------|------|-------------|
102+
| `vout_index` | `compact_size` | 1–9 | Output index. |
103+
| `coin` | `Coin` | variable | Serialized coin (varint-encoded code for height/coinbase, then compressed txout). |
104+
105+
Coins are ordered lexicographically by outpoint (txid, then vout index), matching the LevelDB iteration
106+
order of the coins database.
107+
108+
#### Chunk Merkle Tree
109+
110+
The serialized UTXO set (header + body) is split into chunks of exactly 3,900,000 bytes (3.9 MB). The
111+
last chunk contains the remaining bytes and may be smaller. The chunks form the leaves of a binary
112+
Merkle tree whose root commits to the entire UTXO set.
113+
114+
The leaf hash for each chunk is `SHA256d(chunk_data)`. The tree is built as a balanced binary tree. When
115+
the number of nodes at a level is odd, the last node is promoted unchanged to the next level.
116+
Interior nodes are computed as `SHA256d(left_child || right_child)`.
117+
118+
The leaves are delivered to the node in a single `utxotree` response. A node that knows
119+
the Merkle root for a given UTXO set checks a received list of leaves by recomputing the root and
120+
comparing. The Merkle root is the sole trust input required to verify the integrity of the received UTXO set.
121+
122+
`SHA256d` denotes double-SHA256: `SHA256d(x) = SHA256(SHA256(x))`.
123+
124+
### Messages
125+
126+
#### `getutxotree`
127+
128+
Sent to request the chunk-hash list for a specific UTXO set.
129+
130+
| Field | Type | Size | Description |
131+
|-------|------|------|-------------|
132+
| `block_hash` | `uint256` | 32 | Block hash identifying the requested UTXO set. |
133+
134+
A node that has advertised `NODE_UTXO_SET` and can serve the requested UTXO set MUST respond with
135+
`utxotree`. If the serving node cannot fulfill the request, it MUST NOT respond. The requesting
136+
node SHOULD apply a reasonable timeout and try another peer.
137+
138+
#### `utxotree`
139+
140+
Sent in response to `getutxotree`, delivering the full chunk-hash list along with per-snapshot
141+
metadata.
142+
143+
| Field | Type | Size | Description |
144+
|-------|------|------|-------------|
145+
| `block_hash` | `uint256` | 32 | Block hash this data corresponds to. |
146+
| `version` | `uint16_t` | 2 | Format version of the serialized UTXO set. |
147+
| `data_length` | `uint64_t` | 8 | Total size of the serialized UTXO set in bytes (header + body). |
148+
| `chunk_hashes` | `uint256[]` | 32 × N | The ordered list of N chunk hashes, where N = `ceil(data_length / 3,900,000)`. |
149+
150+
Upon receiving a `utxotree` message, the requesting node MUST recompute the Merkle root from
151+
`chunk_hashes` and compare it against the Merkle root it knows for the corresponding UTXO set. If
152+
the roots do not match, the node MUST discard the response and MUST disconnect the peer.
153+
154+
#### `getutxoset`
155+
156+
Sent to request a single chunk of UTXO set data. The requesting node MUST have received a `utxotree`
157+
for the corresponding UTXO set (from any peer) before sending this message.
158+
159+
| Field | Type | Size | Description |
160+
|-------|------|------|-------------|
161+
| `block_hash` | `uint256` | 32 | Block hash identifying the requested UTXO set. |
162+
| `chunk_index` | `uint32_t` | 4 | Zero-based index of the requested chunk. |
163+
164+
If the serving node cannot fulfill the request, it MUST NOT respond. The requesting node SHOULD apply
165+
a reasonable timeout and try another peer.
166+
167+
#### `utxoset`
168+
169+
Sent in response to `getutxoset`, delivering one chunk.
170+
171+
| Field | Type | Size | Description |
172+
|-------|------|------|-------------|
173+
| `block_hash` | `uint256` | 32 | Block hash this data corresponds to. |
174+
| `chunk_index` | `uint32_t` | 4 | Zero-based index of this chunk. |
175+
| `data` | `bytes` | variable | Chunk payload, exactly 3.9 MB except for the last chunk. |
176+
177+
The transfer is receiver-driven: the requesting node sends one `getutxoset` per chunk. Chunks MAY be
178+
requested in any order and from different peers.
179+
180+
Upon receiving a `utxoset` message, the node MUST compute `SHA256d(data)` and compare it against
181+
`chunk_hashes[chunk_index]` from the `utxotree` it accepted for this UTXO set. If the hashes do not
182+
match, the node MUST discard the chunk and MUST disconnect the peer. A node SHOULD also disconnect
183+
a peer that sends a `utxoset` message with fields (`chunk_index`, `block_hash`) that do not match
184+
the outstanding request.
185+
186+
After all chunks have been received, the node SHOULD parse the reassembled UTXO set against the
187+
serialized UTXO set format to confirm it is well-formed.
188+
189+
### Protocol Flow
190+
191+
1. The requesting node identifies peers advertising `NODE_UTXO_SET`.
192+
2. The requesting node sends `getutxotree` for the desired block hash to one of these peers, or to
193+
several peers to corroborate the Merkle root by agreement if no trusted root is available.
194+
3. The peer or peers respond with `utxotree`. The requesting node verifies each response by
195+
recomputing the Merkle root and comparing it against a value it knows for the given UTXO set,
196+
either from a trusted source or from agreement among multiple peers. A single accepted `utxotree`
197+
can be used as the basis for all subsequent chunk requests for this UTXO set, regardless of
198+
which peer those chunks are fetched from.
199+
4. The requesting node downloads chunks via `getutxoset`/`utxoset` exchanges, verifying each chunk
200+
against its entry in the accepted `utxotree` on receipt. On verification failure the peer is
201+
disconnected and download continues from another peer without losing already-verified chunks.
202+
5. After all chunks are received, the node parses the reassembled UTXO set against the serialized
203+
UTXO set format to confirm that it is well-formed.
204+
205+
Serving nodes are free to limit the number of concurrent and repeated transfers per peer at their own
206+
discretion to manage resource consumption.
207+
208+
## Rationale
209+
210+
**Usage of service bit 14:** Service bits allow selective peer discovery through
211+
DNS seeds and addr relay. Bit 14 is chosen because bits 12 and 13 are reserved by the
212+
Utreexo proposal (BIP 183 draft).
213+
214+
**Direct request model:** Peers signal availability of UTXO sets via the `NODE_UTXO_SET`
215+
service bit; the requesting node identifies the desired UTXO set by block hash when sending
216+
`getutxotree`. The serving node responds only if it can serve that specific UTXO set.
217+
218+
**Per-chunk verification:** The chunk-hash list returned in `utxotree` enables each chunk to be verified
219+
by direct lookup against the accepted list as it arrives, allowing immediate detection of corrupt data,
220+
peer switching without data loss, and parallel download from multiple peers. The list itself is small
221+
(~80 KB for a ~10 GB set). The specified serialization is deterministic, so all honest nodes produce
222+
byte-identical output, guaranteeing Merkle root agreement.
223+
224+
**3.9 MB chunk size:** The number balances round trips (~2,560 for a ~10 GB set) against memory usage
225+
for buffering and verifying a single chunk. Smaller chunks would increase protocol overhead; larger
226+
chunks would increase memory pressure on constrained devices commonly used to run Bitcoin nodes.
227+
Together with the additional message overhead, the `utxoset` message including the chunk data also
228+
sits just below the theoretical maximum block size which means any implementation should be able to
229+
handle messages of this size.
230+
231+
**Reusing the `dumptxoutset` format:** Avoids introducing a new serialization format and ensures
232+
compatibility with UTXO sets already being generated and shared.
233+
234+
**Relationship to BIP 64:** BIP 64 defined a protocol for querying individual UTXOs by outpoint and is
235+
now closed. This BIP addresses a different use case: bulk transfer of the entire UTXO set for node
236+
bootstrapping.
237+
238+
## Backwards Compatibility
239+
240+
This proposal is backward compatible. Peers that do not implement it ignore the new service bit
241+
and never issue the new messages.
242+
243+
## Reference Implementation
244+
245+
[Bitcoin Core implementation pull request](https://github.com/bitcoin/bitcoin/pull/35054)
246+
247+
## Acknowledgements
248+
249+
Thanks to Anthony Towns for suggesting that the requesting node fetch the full chunk-hash list up front
250+
via the `getutxotree`/`utxotree` exchange rather than per-chunk Merkle proofs, using the Merkle root as
251+
the sole trust anchor in place of a separate serialized hash, dropping the redundant `num_chunks` field,
252+
and the deterministic schedule of served heights.
253+
254+
Thanks also to Murch for catching the service-bit collision with the Utreexo proposal, raising
255+
the Merkle tree malleability concern behind promoting odd nodes unchanged rather than duplicating them,
256+
and prompting the Backwards Compatibility section; stickies-v for raising the data-availability concern
257+
that motivated a fixed schedule of served heights; Luke Dashjr for arguing against a separate discovery
258+
step on privacy grounds and suggesting the serialization format version be carried; and Daniela Brozzoni
259+
for helping make the peer-disconnection rules consistent.
260+
261+
## Copyright
262+
263+
This BIP is made available under the terms of the 2-clause BSD license. See
264+
https://opensource.org/license/BSD-2-Clause for more information.
265+
266+
## Changelog
267+
268+
* __0.5.0__ (2026-06-03):
269+
* Defined a deterministic schedule of served heights for the `NODE_UTXO_SET` service bit
270+
* Added Acknowledgements section
271+
* __0.4.0__ (2026-05-18):
272+
* Removed `num_chunks` from `utxotree`
273+
* __0.3.0__ (2026-05-17):
274+
* Moved service bit from 12 to 14 to avoid collision with the Utreexo proposal (BIP 183 draft)
275+
* Changed Merkle tree construction: odd nodes are promoted unchanged rather than duplicated
276+
* __0.2.0__ (2026-05-04):
277+
* Dropped discovery before download approach, instead request the chunk-hash list via `getutxotree`/`utxotree`
278+
* Dropped per-chunk Merkle proofs; chunks verified directly against the chunk-hash list
279+
* Dropped `height` from requests (`block_hash` is the sole identifier); added format `version` to `utxotree`
280+
* Dropped references to the serialized hash; the Merkle root is the sole integrity check
281+
* __0.1.0__ (2026-04-10):
282+
* Initial draft

0 commit comments

Comments
 (0)