Skip to content

Commit 231d1c1

Browse files
frisitanofrisitano
committed
proposal: opt-in witness retrieval via flag on existing engine methods
Adds src/engine/witness-retrieval.md describing an Engine API extension that allows the consensus layer to opt in to receiving the execution witness alongside the existing payload validation and payload production flows. Design summary: - engine_newPayloadV6: extends V5 with an optional `requestWitness` param and an optional `executionWitness` field on PayloadStatusV2. - engine_forkchoiceUpdatedV5: accepts PayloadAttributesV5, which adds a `requestWitness` field so the EL can configure witness collection before execution begins. - engine_getPayloadV7: response gains an optional `executionWitness`, populated when the originating FCU set `requestWitness = true`. - ExecutionWitnessV1: DATA representing SSZ-encoded witness bytes, following the on-wire convention of BlobV1. Container schema defined externally in a consensus-specs companion document. Offered as an alternative to the *WithWitness method approach (issue ethereum#741, PR ethereum#773). Rationale doc compares this against *WithWitness and notes composition with the SSZ-REST transport (PR ethereum#764).
1 parent 8d6b784 commit 231d1c1

2 files changed

Lines changed: 322 additions & 0 deletions

File tree

‎src/engine/witness-retrieval.md‎

Lines changed: 321 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,321 @@
1+
# Engine API -- Witness Retrieval (Proposal)
2+
3+
Engine API extensions that allow the consensus layer to opt in to receiving the
4+
execution witness alongside the existing payload validation and payload
5+
production flows.
6+
7+
This specification is based on and extends [Engine API - Amsterdam](./amsterdam.md).
8+
9+
> **Status:** Design proposal. This document is offered as an alternative to
10+
> introducing dedicated `*WithWitness` methods (cf. issue
11+
> [#741](https://github.com/ethereum/execution-apis/issues/741) and PR
12+
> [#773](https://github.com/ethereum/execution-apis/pull/773)). It is
13+
> intended to be evaluated by Engine API maintainers prior to merging.
14+
15+
## Table of contents
16+
17+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
18+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
19+
20+
- [Motivation](#motivation)
21+
- [Structures](#structures)
22+
- [ExecutionWitnessV1](#executionwitnessv1)
23+
- [PayloadStatusV2](#payloadstatusv2)
24+
- [PayloadAttributesV5](#payloadattributesv5)
25+
- [Methods](#methods)
26+
- [engine_newPayloadV6](#engine_newpayloadv6)
27+
- [engine_getPayloadV7](#engine_getpayloadv7)
28+
- [engine_forkchoiceUpdatedV5](#engine_forkchoiceupdatedv5)
29+
- [Rationale](#rationale)
30+
- [Capabilities and feature gating](#capabilities-and-feature-gating)
31+
32+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
33+
34+
## Motivation
35+
36+
Stateless validation flows require the consensus layer (CL) — or a prover
37+
acting on its behalf — to obtain the **execution witness** for a block:
38+
the set of state and bytecode accesses produced during execution. Two
39+
paths exist for the witness to leave the execution layer (EL):
40+
41+
1. **The CL receives an inbound payload over gossip.** The CL forwards it to the
42+
EL via `engine_newPayload*`, and wants the EL to attach the witness to the
43+
validation response.
44+
2. **The CL drives block production locally.** The CL initiates payload building
45+
via `engine_forkchoiceUpdated*`, retrieves the built payload via
46+
`engine_getPayload*`, and wants the EL to attach the witness to the produced
47+
payload.
48+
49+
The currently-proposed designs (issue [#741][issue-741], PR [#773][pr-773])
50+
introduce parallel `engine_newPayloadV{N}WithWitness` methods that mirror the
51+
existing methods and additionally return the witness. This document proposes an
52+
alternative: **extend the existing methods with an opt-in flag**, and route the
53+
build-side opt-in through `PayloadAttributes`, where similar fork-scoped
54+
extensions (`withdrawals`, `parentBeaconBlockRoot`) have historically been
55+
added.
56+
57+
[issue-741]: https://github.com/ethereum/execution-apis/issues/741
58+
[pr-773]: https://github.com/ethereum/execution-apis/pull/773
59+
60+
## Structures
61+
62+
### ExecutionWitnessV1
63+
64+
`DATA` - Variable-length bytes representing an SSZ-encoded
65+
`ExecutionWitness` container. The container schema is defined by an
66+
accompanying consensus-specs document and is out of scope for this
67+
specification.
68+
69+
This follows the same on-wire convention used elsewhere in the Engine API
70+
for SSZ payloads (cf. the `blob` field in [`BlobV1`](./cancun.md)): a
71+
JSON-RPC `DATA` value whose underlying bytes are SSZ-encoded according to
72+
a consensus-specs definition. The encoding is fixed by this specification,
73+
so no format tag is required; on SSZ-REST transports (cf.
74+
[PR #764](https://github.com/ethereum/execution-apis/pull/764)) the same
75+
bytes are carried directly without hex wrapping.
76+
77+
### PayloadStatusV2
78+
79+
This structure has the syntax of [`PayloadStatusV1`](./paris.md#payloadstatusv1)
80+
and appends a single optional field: `executionWitness`.
81+
82+
- `status`: `String` - One of: `VALID`, `INVALID`, `SYNCING`, `ACCEPTED`,
83+
`INVALID_BLOCK_HASH`.
84+
- `latestValidHash`: `DATA|null`, 32 Bytes - The hash of the most recent valid
85+
block in the branch defined by payload and its ancestors.
86+
- `validationError`: `String|null` - A message providing additional details on
87+
the validation error if the payload is classified as `INVALID`.
88+
- `executionWitness`: [`ExecutionWitnessV1`](#executionwitnessv1)`|null` -
89+
SSZ-encoded execution witness. Present **only if** `requestWitness` was
90+
set to `true` in the associated request **and** `status` is `VALID`.
91+
`null` in all other cases.
92+
93+
### PayloadAttributesV5
94+
95+
This structure has the syntax of [`PayloadAttributesV4`](./amsterdam.md#payloadattributesv4)
96+
and appends a single field: `requestWitness`.
97+
98+
- `timestamp`: `QUANTITY`, 64 Bits - value for the `timestamp` field of the new payload
99+
- `prevRandao`: `DATA`, 32 Bytes - value for the `prevRandao` field of the new payload
100+
- `suggestedFeeRecipient`: `DATA`, 20 Bytes - suggested value for the `feeRecipient` field of the new payload
101+
- `withdrawals`: `Array of WithdrawalV1` - Array of withdrawals, each object is an `OBJECT` containing the fields of a `WithdrawalV1` structure.
102+
- `parentBeaconBlockRoot`: `DATA`, 32 Bytes - Root of the parent beacon block.
103+
- `slotNumber`: `QUANTITY`, 64 Bits - value for the `slotNumber` field of the new payload
104+
- `requestWitness`: `BOOLEAN` - if `true`, the EL **MUST** collect and persist
105+
the execution witness alongside the produced payload so that it can be
106+
returned by a subsequent call to [`engine_getPayloadV7`](#engine_getpayloadv7).
107+
Defaults to `false` when omitted.
108+
109+
## Methods
110+
111+
### engine_newPayloadV6
112+
113+
This method extends [`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5)
114+
with an optional witness-request flag and an optional witness in the response.
115+
116+
#### Request
117+
118+
* method: `engine_newPayloadV6`
119+
* params:
120+
1. `executionPayload`: [`ExecutionPayloadV4`](./amsterdam.md#executionpayloadv4).
121+
2. `expectedBlobVersionedHashes`: `Array of DATA`, 32 Bytes - Array of expected blob versioned hashes to validate.
122+
3. `parentBeaconBlockRoot`: `DATA`, 32 Bytes - Root of the parent beacon block.
123+
4. `executionRequests`: `Array of DATA` - List of execution layer triggered requests.
124+
5. `requestWitness`: `BOOLEAN` - if `true`, the EL **MUST** include the
125+
execution witness in the response when the payload is `VALID`.
126+
Defaults to `false` when omitted.
127+
* timeout: 8s
128+
129+
#### Response
130+
131+
* result: [`PayloadStatusV2`](#payloadstatusv2)
132+
* error: code and message set in case an exception happens while processing the
133+
payload.
134+
135+
#### Specification
136+
137+
This method follows the same specification as
138+
[`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5) with the following
139+
additions:
140+
141+
1. If `requestWitness` is `true` **and** the payload validates to `VALID`, the
142+
EL **MUST** populate `result.executionWitness` with an
143+
[`ExecutionWitnessV1`](#executionwitnessv1) object reflecting the state and
144+
bytecode accesses produced during execution.
145+
146+
2. If `requestWitness` is `true` **and** validation does not result in `VALID`
147+
(i.e. status is `INVALID`, `SYNCING`, `ACCEPTED`, or `INVALID_BLOCK_HASH`),
148+
the EL **MUST** set `result.executionWitness` to `null`.
149+
150+
3. If `requestWitness` is `false` or omitted, the EL **MUST** set
151+
`result.executionWitness` to `null` and **MUST NOT** perform additional work
152+
to collect the witness on the critical path.
153+
154+
4. Witness collection **MUST NOT** affect the validation outcome reported
155+
in `result.status`.
156+
157+
5. Clients that have not advertised the `engine_newPayloadV6` capability via
158+
capabilities exchange **MUST** continue to be addressed via
159+
[`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5).
160+
161+
### engine_getPayloadV7
162+
163+
This method extends [`engine_getPayloadV6`](./amsterdam.md#engine_getpayloadv6)
164+
with an optional witness in the response. The witness is returned only when
165+
the corresponding payload-build call (`engine_forkchoiceUpdatedV5`) was issued
166+
with `payloadAttributes.requestWitness = true`.
167+
168+
#### Request
169+
170+
* method: `engine_getPayloadV7`
171+
* params:
172+
1. `payloadId`: `DATA`, 8 Bytes - Identifier of the payload build process
173+
* timeout: 1s
174+
175+
#### Response
176+
177+
* result: `object`
178+
- `executionPayload`: [`ExecutionPayloadV4`](./amsterdam.md#executionpayloadv4)
179+
- `blockValue` : `QUANTITY`, 256 Bits - The expected value to be received by the `feeRecipient` in wei
180+
- `blobsBundle`: [`BlobsBundleV2`](./osaka.md#blobsbundlev2) - Bundle with data corresponding to blob transactions included into `executionPayload`
181+
- `shouldOverrideBuilder` : `BOOLEAN` - Suggestion from the execution layer to use this `executionPayload` instead of an externally provided one
182+
- `executionRequests`: `Array of DATA` - Execution layer triggered requests obtained from the `executionPayload` transaction execution.
183+
- `executionWitness`: [`ExecutionWitnessV1`](#executionwitnessv1)`|null` -
184+
SSZ-encoded execution witness. Present **only if** the originating
185+
`engine_forkchoiceUpdatedV5` call set
186+
`payloadAttributes.requestWitness = true`. `null` otherwise.
187+
* error: code and message set in case an exception happens while getting the payload.
188+
189+
#### Specification
190+
191+
This method follows the same specification as
192+
[`engine_getPayloadV6`](./amsterdam.md#engine_getpayloadv6) with the following
193+
additions:
194+
195+
1. If the build was initiated with `payloadAttributes.requestWitness = true`,
196+
the EL **MUST** populate `result.executionWitness` with an
197+
[`ExecutionWitnessV1`](#executionwitnessv1) reflecting the witness collected
198+
during payload building.
199+
200+
2. If the build was initiated with `payloadAttributes.requestWitness = false`
201+
(or omitted), the EL **MUST** set `result.executionWitness` to `null` and
202+
**MUST NOT** perform additional work to collect the witness retroactively.
203+
204+
### engine_forkchoiceUpdatedV5
205+
206+
This method extends [`engine_forkchoiceUpdatedV4`](./amsterdam.md#engine_forkchoiceupdatedv4)
207+
to accept [`PayloadAttributesV5`](#payloadattributesv5).
208+
209+
#### Request
210+
211+
* method: `engine_forkchoiceUpdatedV5`
212+
* params:
213+
1. `forkchoiceState`: [`ForkchoiceStateV1`](./paris.md#ForkchoiceStateV1).
214+
2. `payloadAttributes`: `Object|null` - Instance of [`PayloadAttributesV5`](#payloadattributesv5) or `null`.
215+
3. `custodyColumns`: `DATA|null`, 16 Bytes - Interpreted as a bitarray of length `CELLS_PER_EXT_BLOB` indicating which column indices form the CL's custody set, or `null` if the CL does not provide custody services.
216+
* timeout: 8s
217+
218+
#### Response
219+
220+
Refer to the response for [`engine_forkchoiceUpdatedV4`](./amsterdam.md#engine_forkchoiceupdatedv4).
221+
222+
#### Specification
223+
224+
This method follows the same specification as
225+
[`engine_forkchoiceUpdatedV4`](./amsterdam.md#engine_forkchoiceupdatedv4) with
226+
the following changes:
227+
228+
1. Validate `payloadAttributes` against [`PayloadAttributesV5`](#payloadattributesv5)
229+
in place of `PayloadAttributesV4`. If `payloadAttributes` is provided and
230+
does not match this structure, return `-38003: Invalid payload attributes`.
231+
232+
2. If `payloadAttributes.requestWitness` is `true`, the EL **MUST** configure
233+
the payload-build job initiated by this call to collect and persist the
234+
execution witness, such that a subsequent
235+
[`engine_getPayloadV7`](#engine_getpayloadv7) call addressing the resulting
236+
`payloadId` can return it.
237+
238+
3. If `payloadAttributes.requestWitness` is `false` or omitted, the EL
239+
**MUST NOT** perform additional work to collect the witness for this build
240+
job.
241+
242+
## Rationale
243+
244+
### Why a flag, not a parallel method
245+
246+
Engine API methods version once per fork (`engine_newPayloadV1` … `V5`). The
247+
`*WithWitness` design proposed in PR [#773][pr-773] doubles this surface,
248+
producing a `V{N}` and a `V{N}WithWitness` per fork going forward. Each
249+
variant must be specified, openrpc-modeled, capabilities-negotiated, and
250+
client-implemented in lockstep with the standard methods.
251+
252+
Routing the opt-in through a single boolean flag on the existing methods:
253+
254+
- Keeps the method count constant across forks.
255+
- Avoids the response-shape drift between `V{N}` and `V{N}WithWitness` that
256+
recurs every fork.
257+
- Composes naturally with the SSZ-REST transport proposed in
258+
PR [#764](https://github.com/ethereum/execution-apis/pull/764): the same
259+
flag governs whether the SSZ response includes a witness field, with no
260+
additional REST endpoint needed.
261+
262+
### Why route the build-side opt-in through `PayloadAttributes`
263+
264+
Witness collection on the build path **must be decided before execution
265+
begins**, because the EL needs to instrument trie traversal during execution
266+
to record accesses. `engine_getPayloadV{N}` is called *after* execution has
267+
finished and the block is built; adding a flag there would either force the
268+
EL to always collect witnesses (defeating the opt-in), or trigger re-execution
269+
(unacceptable on the critical path).
270+
271+
`PayloadAttributes` is the canonical extension point for build-side knobs and
272+
has accumulated similar fork-scoped fields previously: `withdrawals` (Shapella),
273+
`parentBeaconBlockRoot` (Cancun), `slotNumber` (Amsterdam). Adding
274+
`requestWitness` here is consistent with that precedent and gives the EL the
275+
information it needs at build-job initiation time.
276+
277+
### Why bump `PayloadStatus` to V2
278+
279+
The Engine API convention bumps a structure's `V` whenever its shape changes
280+
(`ExecutionPayloadV1`…`V4`, `PayloadAttributesV1`…`V5`, `BlobsBundleV1`…`V2`).
281+
Adding `executionWitness` to the response shape is a structural change, so
282+
`PayloadStatusV2` follows that pattern. `engine_newPayloadV1`…`V5` continue to
283+
return `PayloadStatusV1` unchanged; only `engine_newPayloadV6` returns the new
284+
type. This localises the change to a single method version and leaves all
285+
existing openrpc schemas untouched.
286+
287+
### Considered alternatives
288+
289+
1. **`engine_newPayloadV{N}WithWitness` (PR #773).** Proven design and already
290+
has a working prototype, but doubles method surface and creates a
291+
redundant verb once SSZ-REST lands (acknowledged by the PR author in
292+
prior discussion).
293+
294+
2. **Always return the witness.** Simplest schema, but forces witness
295+
collection on every block — a meaningful CPU and I/O cost (~574 ms
296+
witness-generation step measured by PR #773's benchmarks) that most CLs
297+
do not need.
298+
299+
3. **Capabilities-only gating with no flag.** Use the engine_exchangeCapabilities
300+
handshake alone to decide whether to attach the witness. Too coarse: a CL
301+
may want the witness for some blocks (e.g. when acting as a stateless
302+
prover) and not others.
303+
304+
4. **Trailing optional param on `engine_newPayloadV5` without a version
305+
bump.** Backward-compatible at the JSON-RPC level, but breaks the
306+
convention that every shape change bumps `V`. Rejected for consistency
307+
with existing fork-aligned versioning.
308+
309+
## Capabilities and feature gating
310+
311+
The `engine_newPayloadV6`, `engine_forkchoiceUpdatedV5`, and
312+
`engine_getPayloadV7` methods **MUST** be advertised through
313+
`engine_exchangeCapabilities`. CLs **MUST NOT** invoke these methods without
314+
prior capability confirmation from the EL. An EL that does not yet support
315+
witness retrieval continues to be addressed via the V5 / V4 / V6 predecessors
316+
unmodified.
317+
318+
Witness-retrieval support is independent of any specific fork activation:
319+
once advertised, an EL may serve witnesses for any post-Amsterdam block
320+
the CL requests them for. Because the flag is opt-in per call, the EL
321+
incurs witness-collection cost only when a CL asks for it.

‎wordlist.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ eth
3636
ethereum
3737
ethereumjs
3838
EVM
39+
executionwitnessv
3940
exitv
4041
feeRecipient
4142
forkchoiceupdatedresponsev

0 commit comments

Comments
 (0)