feat(consume): add `ExceptionMapper` support to the consume-engine simulator by marioevz · Pull Request #1416 · ethereum/execution-spec-tests

marioevz · 2025-04-09T00:25:52Z

🗒️ Description

Adds support of the exception mapper to consume-engine:

consume engine now checks exceptions returned by the execution clients in their Engine API responses, specifically in the validationErrorfield of the engine_newPayloadVX method.

Exception mappers of all current mainnet clients have been added to the ethereum_clis library.

The feature can be disabled by using --disable-strict-exception-matching for specific clients or forks.

🔗 Related Issues

✅ Checklist

All: Set appropriate labels for the changes.
All: Considered squashing commits to improve commit history.
All: Added an entry to CHANGELOG.md.
All: Considered updating the online docs in the ./docs/ directory.
Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

winsvega · 2025-04-14T10:42:19Z

I found there are some issue with refactored exception mapper. perhaps this can be address in here in this PR
if not fixed already (some)
ethereum/execution-specs#1538

winsvega

waut the complexity of exception mapper is growing. I need to study python again.

danceratopz

Love this, few suggestions, also in #1464.

danceratopz

LGTM!

…only affects generated block tests)

Temporarily increase failure threshold. Failures because of mismatch in error mappings or sequence. Some of these are nice to have but not urgent: ethereum/execution-spec-tests#1416

…mulator (ethereum#1416) * fix(exceptions): UndefinedException return on new * feat(rpc,plugins/consume): Support exception mapper * fix(exceptions): Allow undefined mapper * feat(plugins/consume): Add more mappers * fix(logging): don't use newlines within log messages * refactor(exceptions,specs): ExceptionWithMessage contains a list * new(exceptions): Add `INVALID_BLOCK_HASH` exception * feat(clis): Add ethjs exceptions * feat(clis): Add geth exceptions * feat(plugins/consume): Add besu exceptions * feat(plugins/consume): Add nethermind exceptions * feat(plugins/consume): Add reth exceptions * feat(consume): add `--strict-exception-matching` parameter * more reth exceptions * refactor mappers * add ethjs exception * fix(tests): EIP-7685: Add expected exception to `test_invalid_multi_type_requests_engine` * add reth exception * fix(plugins/consume): log failure * feat(clis): Add Erigon Exception Mapper * feat(specs): Allow setting expected block exception for state tests (only affects generated block tests) * fix(tests): Add block exceltion to test_blob_type_tx_pre_fork * fix(clis): Exceptions in mappers * change(plugins/consume): Enable strict exception matching by default * fix(specs): Exception message check on strict match * docs: changelog * chore(clis): update exception mappers for TYPE_4_TX_PRE_FORK * refactor(consume): cache exception mappers in session-scoped fixture * refactor(consume): make string match more robust by applying `lower()` * feat(consume): allow disabling of strict exception checking per fork * refactor(consume/hive): create `strict_exception_matching` fixture * fix: whitelist * changelog: Add exception * changelog: Invalid versioned hashes exception * refactor(consume/engine): Better error logging * fix(clis/reth): Exception mapper * fix(clis/besu): Exception mapper * fix(clis/geth): Exception mapper * fix(clis/ethereumjs): Exception mapper * fix(clis/erigon): Exception mapper --------- Co-authored-by: danceratopz <danceratopz@gmail.com>

danceratopz changed the title ~~feat(plugins/consume): Support Exception Mapper~~ feat(consume): add ExceptionMapper support to the consume-engine simulator Apr 11, 2025

danceratopz added type:feat type: Feature scope:consume Scope: Consume command suite labels Apr 11, 2025

spencer-tb mentioned this pull request Apr 11, 2025

feat(release): tasks for version v4.3.0 (Vltava) #1418

Closed

17 tasks

jochem-brouwer reviewed Apr 13, 2025

View reviewed changes

Comment thread src/ethereum_clis/clis/ethereumjs.py

spencer-tb added the fork:prague Prague hardfork label Apr 14, 2025

winsvega reviewed Apr 14, 2025

View reviewed changes

Comment thread src/ethereum_test_exceptions/exception_mapper.py

winsvega reviewed Apr 14, 2025

View reviewed changes

Comment thread src/ethereum_test_exceptions/exception_mapper.py

winsvega reviewed Apr 14, 2025

View reviewed changes

Comment thread src/pytest_plugins/consume/hive_simulators/conftest.py Outdated

winsvega reviewed Apr 14, 2025

View reviewed changes

marioevz mentioned this pull request Apr 14, 2025

Interop Testing #33 | April 14 2025 ethereum/pm#1450

Closed

marioevz force-pushed the exception-mapper-consume branch from f1c23d2 to 5427517 Compare April 14, 2025 19:33

This was referenced Apr 16, 2025

feat(tests): test that 7702 set code txs are rejected pre-prague #1463

Merged

Suggestions for: feat(consume): add ExceptionMapper support to the consume-engine simulator #1464

Merged

danceratopz reviewed Apr 16, 2025

View reviewed changes

marioevz marked this pull request as ready for review April 16, 2025 17:07

marioevz force-pushed the exception-mapper-consume branch from 27845aa to ce47dbb Compare April 16, 2025 23:53

danceratopz approved these changes Apr 17, 2025

View reviewed changes

marioevz and others added 11 commits April 17, 2025 15:12

fix(exceptions): UndefinedException return on new

ede149b

feat(rpc,plugins/consume): Support exception mapper

eae3ebf

fix(exceptions): Allow undefined mapper

42a2ffd

feat(plugins/consume): Add more mappers

44848a0

fix(logging): don't use newlines within log messages

750ac8f

refactor(exceptions,specs): ExceptionWithMessage contains a list

612b70d

new(exceptions): Add INVALID_BLOCK_HASH exception

71a5ba6

feat(clis): Add ethjs exceptions

dee4312

feat(clis): Add geth exceptions

de2c2e6

feat(plugins/consume): Add besu exceptions

5830a42

feat(plugins/consume): Add nethermind exceptions

e96339e

marioevz and others added 15 commits April 17, 2025 15:13

feat(clis): Add Erigon Exception Mapper

922e4cd

feat(specs): Allow setting expected block exception for state tests (…

1eba8c5

…only affects generated block tests)

fix(tests): Add block exceltion to test_blob_type_tx_pre_fork

6913aac

fix(clis): Exceptions in mappers

2f6a05f

change(plugins/consume): Enable strict exception matching by default

a1a2752

fix(specs): Exception message check on strict match

23296c9

docs: changelog

de60efb

chore(clis): update exception mappers for TYPE_4_TX_PRE_FORK

fe57dd3

refactor(consume): cache exception mappers in session-scoped fixture

aa07b4d

refactor(consume): make string match more robust by applying lower()

5067df6

feat(consume): allow disabling of strict exception checking per fork

7988a7e

refactor(consume/hive): create strict_exception_matching fixture

6c9628b

fix: whitelist

4d574d8

changelog: Add exception

bea0594

changelog: Invalid versioned hashes exception

4963923

marioevz force-pushed the exception-mapper-consume branch from 35239db to 4963923 Compare April 17, 2025 16:19

marioevz mentioned this pull request Oct 9, 2025

Refactor Exception Mapper to disallow duplicated Keys ethereum/execution-specs#1542

Open

marioevz added 6 commits April 17, 2025 17:22

refactor(consume/engine): Better error logging

f6b1d9f

fix(clis/reth): Exception mapper

0d9d8b5

fix(clis/besu): Exception mapper

696e2e9

fix(clis/geth): Exception mapper

deb8a3f

fix(clis/ethereumjs): Exception mapper

a83d867

fix(clis/erigon): Exception mapper

4b50b15

marioevz merged commit cb4ccde into main Apr 17, 2025
22 checks passed

marioevz deleted the exception-mapper-consume branch April 17, 2025 18:53

somnergy mentioned this pull request Apr 24, 2025

workflows: increase hive eest threshold temporarily erigontech/erigon#14728

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(consume): add `ExceptionMapper` support to the consume-engine simulator#1416

feat(consume): add `ExceptionMapper` support to the consume-engine simulator#1416
marioevz merged 40 commits into
mainfrom
exception-mapper-consume

marioevz commented Apr 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

winsvega commented Apr 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

winsvega left a comment

Uh oh!

danceratopz left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danceratopz left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

marioevz commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🗒️ Description

🔗 Related Issues

✅ Checklist

Uh oh!

Uh oh!

winsvega commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

winsvega left a comment

Choose a reason for hiding this comment

Uh oh!

danceratopz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danceratopz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

marioevz commented Apr 9, 2025 •

edited

Loading

winsvega commented Apr 14, 2025 •

edited

Loading