Skip to content

Commit a9abd46

Browse files
danceratopzdanceratopz
authored
docs(test-consume): fix and augment enginex docs (ethereum#3143)
1 parent 0f8b81b commit a9abd46

1 file changed

Lines changed: 36 additions & 15 deletions

File tree

‎docs/running_tests/running.md‎

Lines changed: 36 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Both `consume` and `execute` provide sub-commands which correspond to different
1414
| [`consume direct`](#direct) | Client consume tests via a `statetest` interface | EVM | None | Module test |
1515
| [`consume direct`](#direct) | Client consume tests via a `blocktest` interface | EVM, block processing | None | Module test,</br>Integration test |
1616
| [`consume engine`](#engine) | Client imports blocks via Engine API `EngineNewPayload` in Hive | EVM, block processing, Engine API | Staging, Hive | System test |
17-
| [`consume enginex`](#enginex) | Client imports blocks via Engine API in Hive, optimized by client reuse | EVM, block processing, Engine API | Staging, Hive | System test |
17+
| [`consume enginex`](#enginex) | Client imports blocks via Engine API in Hive, optimized by client reuse | EVM, block processing, Engine API, chain reorgs (implicit\*\*) | Staging, Hive | System test |
1818
| [`consume sync`](#sync) | Client syncs from another client using Engine API in Hive | EVM, block processing, Engine API, P2P sync | Staging, Hive | System test |
1919
| [`consume rlp`](#rlp) | Client imports RLP-encoded blocks upon start-up in Hive | EVM, block processing, RLP import (sync\*) | Staging, Hive | System test |
2020
| [`build-block`](#block-building) | Client builds blocks via `testing_buildBlockV1` in Hive, validated against fixture | EVM, block production, Engine API (testing namespace) | Staging, Hive | System test |
@@ -23,6 +23,8 @@ Both `consume` and `execute` provide sub-commands which correspond to different
2323

2424
\*sync: Depending on code paths used in the client implementation, see the [RLP vs Engine Simulator section below](#engine-vs-rlp-simulator).
2525

26+
\*\*chain reorgs: A side-effect of client reuse, not something the test cases describe, see the [Implicit Chain Reorg Coverage section below](#implicit-chain-reorg-coverage).
27+
2628
The following sections describe the different methods in more detail.
2729

2830
!!! note "`./hive --sim=eels/consume-engine` vs `consume engine`"
@@ -59,10 +61,12 @@ The `consume engine` command:
5961

6062
1. **Initializes the execution client** with genesis state.
6163
2. **Connects via Engine API** (port 8551), primitively mocking a consensus client.
62-
3. **Sends a forkchoice update** to establish the chain head.
63-
4. **Submits payloads** using `engine_newPayload` calls.
64-
5. **Validates responses** against expected results.
65-
6. **Tests error conditions** and exception handling.
64+
3. **Sends a forkchoice update** to the genesis block to establish the chain head.
65+
4. **Verifies the client's genesis block hash** via `eth_getBlockByNumber(0)`.
66+
5. **Submits payloads** using `engine_newPayload` calls.
67+
6. **Validates responses** against expected results.
68+
7. **Sends a forkchoice update** after each valid payload to advance the chain head.
69+
8. **Tests error conditions** and exception handling.
6670

6771
## EngineX
6872

@@ -78,29 +82,46 @@ The `consume enginex` command, for each pre-allocation group:
7882

7983
1. **Initializes the execution client** with the group's shared genesis state.
8084
2. **Connects via Engine API** (port 8551).
81-
3. **Executes all tests in the group** against the same client:
85+
3. **Executes all tests in the group** against the same client. Each test:
8286

83-
- Submits payloads from each test using `engine_newPayload` calls.
87+
- Sends a forkchoice update to the genesis block, resetting the chain head.
88+
- Verifies the client's genesis block hash via `eth_getBlockByNumber(0)`; this is only done for the first test executed against the client, as genesis is immutable.
89+
- Submits payloads from the test using `engine_newPayload` calls.
8490
- Validates responses against expected results.
91+
- Sends a forkchoice update after each valid payload to advance the chain head.
8592
- Tests error conditions and exception handling.
8693

8794
4. **Stops the client** when all tests in the group complete.
8895

96+
### Implicit Chain Reorg Coverage
97+
98+
Client reuse gives `consume enginex` coverage that `consume engine` does not have. The forkchoice update at the start of each test resets the client's head from the previous test's chain tip back to genesis. The payload that follows is therefore a sibling of a block the client already imported and considered canonical (the same parent and block number, but a different block hash), and the forkchoice update sent after it makes the new branch canonical.
99+
100+
Every test after the first in a pre-allocation group consequently exercises the client's chain reorganization path: rolling the head state back to an ancestor, importing a competing block at an already-occupied height, and re-canonicalizing a new branch.
101+
102+
!!! note "This coverage is implicit"
103+
104+
No `blockchain_test_engine_x` fixture describes a reorg; the reorgs are an artifact of how the simulator reuses clients. A test whose payloads are all invalid also leaves the head at genesis, so no rollback precedes the next test in the group.
105+
106+
It does mean, however, that a test which fails under `consume enginex` but passes under `consume engine` is more likely to indicate a bug in the client's reorg, head state rollback or block caching logic than in its EVM or block validation logic.
107+
89108
### Engine vs EngineX
90109

91-
| | `consume engine` | `consume enginex` |
92-
| -------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------ |
93-
| **Fixture format** | [`blockchain_test_engine`](./test_formats/blockchain_test_engine.md) | [`blockchain_test_engine_x`](./test_formats/blockchain_test_engine_x.md) |
94-
| **Client lifecycle** | New client per test | Client reused across tests with same pre-alloc |
95-
| **Fork choice update** | FCU called for genesis and final payload | FCU for genesis and final payload skipped |
96-
| **Execution speed** | Slower (client startup overhead) | Faster (amortized startup cost) |
97-
| **Test isolation** | Full isolation | Shared genesis state within group |
110+
| | `consume engine` | `consume enginex` |
111+
| ----------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
112+
| **Fixture format** | [`blockchain_test_engine`](./test_formats/blockchain_test_engine.md) | [`blockchain_test_engine_x`](./test_formats/blockchain_test_engine_x.md) |
113+
| **Client lifecycle** | New client per test | Client reused across tests with same pre-alloc |
114+
| **Engine API flow** | FCU to genesis, then an `engine_newPayload` and FCU per valid payload | Identical, to keep both methods equivalent |
115+
| **Genesis block check** | `eth_getBlockByNumber(0)` per test | `eth_getBlockByNumber(0)` once per client; genesis is immutable |
116+
| **Execution speed** | Slower (client startup overhead) | Faster (amortized startup cost) |
117+
| **Test isolation** | Full isolation | Shared client and genesis state within group; the chain head is reset to genesis for each test |
118+
| **Chain reorgs** | Not exercised; each client executes one test's payloads only | [Implicitly exercised](#implicit-chain-reorg-coverage) by every test after the first in a group |
98119

99120
EngineX achieves faster execution by:
100121

101122
1. **Grouping tests** by their pre-allocation state (genesis configuration).
102123
2. **Reusing clients** across all tests in a group, avoiding repeated client startup.
103-
3. **Skipping redundant initialization** since the client is already at the expected genesis state.
124+
3. **Skipping the redundant genesis block check** for reused clients: the client's genesis block hash is verified once per client, instead of once per test.
104125

105126
!!! note "When to use EngineX vs Engine"
106127

0 commit comments

Comments
 (0)