feat: add ssz to engine api#764
Conversation
…eflect optionality removal Removes `Optional[T]` mapping as it is replaced by specific zero/empty value encoding in container definitions. Updates `PayloadStatusV1`, `ForkchoiceUpdatedResponseV1`, `ExecutionPayloadBodyV1`, and `ExecutionPayloadBodyV2` to use non-optional types where JSON mapping implies a zero/empty value instead of true optionality. Updates SSZ mappings for `engine_getPayloadBodiesByHashV1/V2` and `engine_getBlobsV1/V2/V3` to use nested lists (`List[List[T, 1]]`) instead of `Optional[T]` to represent the presence or absence of data, consistent with non-null SSZ encoding for absent data. Adds notes explaining the zero/empty encoding for absent fields.
…update examples The documentation for how `T or null` maps to SSZ encoding is being clarified. Replaced the generic statement about `Optional[T]` being encoded as `List[T, 1]` with a more direct explanation that it is represented as `List[T, 1]`. Also updated the description for `payload_attributes` in `ForkchoiceUpdated` requests to explicitly state that presence is indicated by a list with 1 element, matching the SSZ type definition. Additionally, added missing vocabulary words to `wordlist.txt` to improve future documentation generation tools.
…ation to reflect capability exchange The transport negotiation mechanism has been updated to exclusively use `engine_exchangeCapabilities` over JSON-RPC for determining support of SSZ REST endpoints. This change clarifies the required steps for clients to discover and utilize the binary SSZ transport.
Update documentation to better explain how JSON-RPC remains the default for negotiation and fallback, explicitly stating when binary SSZ is used. This clarifies the steps involved for CL and EL during initialization.
| - The CL uses SSZ natively, forcing a round-trip conversion (SSZ to JSON, then JSON to internal types) at the Engine API boundary. | ||
|
|
||
| Binary SSZ eliminates all of this. The CL sends raw SSZ bytes over HTTP; the EL deserializes directly. No hex encoding, no JSON parsing, no intermediate representations. Payload sizes are reduced by 50% or more compared to JSON-RPC, and serialization is no longer a bottleneck in the critical path between CL and EL. |
There was a problem hiding this comment.
The EL uses RLP, why not RLP? The EL does not currently support SSZ while the CL does support RLP for various reasons. This would reduce the number of libraries in the EL but be net zero for the CL.
What unique utility does SSZ provide?
There was a problem hiding this comment.
The CL does not support RLP, some clients do, but it is unnecessary, it is an optimization.
| | `Array of T` | `List[T, MAX_LENGTH]` (context-dependent) | | ||
| | `T or null` | `List[T, 1]` | | ||
|
|
||
| Nullable types are represented as `List[T, 1]` in SSZ encoding. An empty list (0 elements) denotes absence (`null`). A list with one element denotes presence. |
There was a problem hiding this comment.
Here we mentioned a nullable type is encoded as List[T, 1], but later in the doc some actually nullable values are not encoded this way;
- latest_valid_hash
- payload_id
- block_access_list
So either we adjust this mapping rule or adjust the type declarations below.
Current decelerations of those fields e.g. Byte32 is a sentinel termination for absence, means zero hash 0x000000... not null, but in current JSON it is actually null.
|
|
||
| ### Binary SSZ transport | ||
|
|
||
| Clients **MAY** support a binary SSZ transport as an alternative to JSON-RPC. The binary transport uses resource-oriented REST endpoints with raw SSZ request and response bodies (`application/octet-stream`), eliminating JSON and hex-encoding overhead for fast CL-EL communication. Endpoints follow Beacon API conventions with path-based versioning (e.g., `POST /engine/v5/payloads`). |
There was a problem hiding this comment.
Current specs define the capabilities as JSON-RPC methods names like:
- engine_newPayloadV2
- engine_getPayloadV5
This suggestion change the capabilities vocabulary to strings like POST /engine/v5/payloads.
This is protocol change, not just a transport addition.
|
|
||
| --- | ||
|
|
||
| #### `POST /engine/v1/capabilities` — Exchange capabilities |
There was a problem hiding this comment.
Related to earlier point, If we want to add this endpoint then we should remove engine_exchangeCapabilities reference from the documents and that will be a protocol change.
Or we keep both but must keep the return format of both to one canonical format, and that will existing response format of the engine_exchangeCapabilities to avoid a breaking change to protocol.
| ### Client errors | ||
|
|
||
| | Status | Meaning | Usage | | ||
| | - | - | - | | ||
| | `400` | Bad Request | Malformed SSZ encoding | | ||
| | `401` | Unauthorized | Missing or invalid JWT token | | ||
| | `404` | Not Found | Unknown payload ID | | ||
| | `409` | Conflict | Invalid forkchoice state | | ||
| | `413` | Request Too Large | Request exceeds maximum element count | | ||
| | `422` | Unprocessable Entity | Invalid payload attributes | | ||
|
|
||
| ### Server errors | ||
|
|
||
| | Status | Meaning | Usage | | ||
| | - | - | - | | ||
| | `500` | Internal Server Error | Unexpected server error | | ||
|
|
||
| Error responses use `Content-Type: text/plain` with a human-readable error message body. |
There was a problem hiding this comment.
The existing Engine API has meaningful machine-readable error codes:
- unknown payload
- invalid forkchoice state
- invalid attributes
- too large request
- unsupported fork
The SSZ HTTP version partly maps these, but not completely:
- unsupported fork is missing
- invalid params vs malformed SSZ are not fully separated
- text/plain error bodies are not machine-stable
Would suggest a normative mapping from JSON-RPC error codes to HTTP status + a small structured error body, even if the success path stays raw SSZ.
|
|
||
| Retrieve an execution payload previously requested via forkchoice update with payload attributes. The `{payload_id}` path parameter is the hex-encoded `Bytes8` payload identifier (e.g., `0x1234567890abcdef`). | ||
|
|
||
| This is a safe, idempotent GET operation. The EL may continue optimizing the payload until the slot deadline. |
There was a problem hiding this comment.
This resource is explicitly mutable until the slot deadline, so should not be declared as idempotent.
That means the same URL can legitimately return different bytes over time. If any proxy/middleware/cache gets between CL and EL, stale payload serving becomes possible.
I’d strongly suggest one of:
- mandate Cache-Control: no-store on this endpoint
- or keep it as POST to avoid accidental caching assumptions
| | `MAX_BYTES_PER_TRANSACTION` | `2**30` (1,073,741,824) | [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844) | | ||
| | `MAX_TRANSACTIONS_PER_PAYLOAD` | `2**20` (1,048,576) | [Bellatrix](https://github.com/ethereum/consensus-specs/blob/dev/specs/bellatrix/beacon-chain.md) | | ||
| | `MAX_WITHDRAWALS_PER_PAYLOAD` | `2**4` (16) | [Capella](https://github.com/ethereum/consensus-specs/blob/dev/specs/capella/beacon-chain.md) | |
There was a problem hiding this comment.
These are enormous maximas limits, can easily become victim of DoS.
An SSZ decoder facing malicious lengths/offsets can still be forced into nasty allocation or scan behavior before semantic rejection if the implementation is careless.
We should explicitly mention somewhere in this doc, so all clients should follow it strictly.:
- reject by Content-Length before body read when possible
- stream/offset-validate before allocation
- endpoint-specific maximum body sizes should be enforced operationally
|
|
||
| When a new fork introduces a new method version, a new versioned endpoint is added. Older versioned endpoints **MAY** be deprecated but **SHOULD** remain available for backwards compatibility. | ||
|
|
||
| ### Negotiation and fallback |
There was a problem hiding this comment.
Negotiation says “if both advertise, use SSZ”, but what if:
- EL advertises endpoint support but returns 404/415/500
- one endpoint version is implemented incorrectly while others work
Do we:
- permanently downgrade the whole transport?
- downgrade only that method version?
- retry JSON-RPC immediately?
Without clear rules for negotiation and fallback behavior for each client may diverge.
…llable types
- Drop misleading "idempotent" claim on GET /payloads/{payload_id} and
require Cache-Control: no-store; the payload mutates until the slot
deadline so caches/intermediaries must not store or revalidate it.
- Expand security considerations with explicit DoS guidance: pre-read
Content-Length rejection, length/offset validation before allocation,
and operationally enforced per-endpoint body caps. The protocol-level
maxima bound on-chain validity, not per-request resource use.
- Encode truly-nullable fields per the documented List[T, 1] rule:
PayloadStatusV1.latest_valid_hash, ForkchoiceUpdatedResponseV1.payload_id,
and ExecutionPayloadBodyV2.block_access_list. Restores parity with the
JSON spec (each is non-required / oneOf null) and removes the
zero-sentinel ambiguity. Example response length updated 37 -> 41.
| | `Content-Type` (request) | `application/octet-stream` | SSZ-encoded request container | | ||
| | `Content-Type` (response) | `application/octet-stream` | SSZ-encoded response (success) | | ||
| | `Content-Type` (response) | `text/plain` | Human-readable error message | | ||
| | `Accept` (request) | `application/octet-stream` | Client accepts SSZ-encoded responses | |
There was a problem hiding this comment.
I would also like that we support both application/json and application/octet-stream. The application/json would just the existing JSON-RPC with layer with REST interface. That will be very easy for everyone to implement. Later we can add application/octet-stream support for individual endpoint.
|
The motivation for binary SSZ is clear and the performance gains it promises, especially with blobs, are significant and necessary for the protocol's evolution. However, I have some concerns about coupling the transport layer migration (JSON-RPC to REST) with the encoding semantics change (JSON/hex to SSZ). I propose a staged approach that separates these concerns, allowing for a more robust and predictable transition for CL and EL clients. Core Recommendation: Implement REST Transport with Current JSON Semantics First, Then Introduce SSZ Gradually. This approach involves two main phases: Phase 1: Introduce REST Transport with Existing JSON Semantics
Phase 2: Introduce Optional Binary SSZ Encoding Gradually, Endpoint-by-Endpoint
This phased approach embodies the principle of "slow is smooth, smooth is fast" for protocol evolution. It allows implementers to digest one architectural change at a time, leading to a more stable and predictable ecosystem. |
4 participants
Core change: Full binary SSZ over REST. No JSON, no hex encoding - raw SSZ bytes over HTTP.
Rationale: