Skip to content

Update to conformance test suite#2742

Merged
automergerpr-permission-manager[bot] merged 10 commits into
masterfrom
conformance-tests-update
Apr 17, 2025
Merged

Update to conformance test suite#2742
automergerpr-permission-manager[bot] merged 10 commits into
masterfrom
conformance-tests-update

Conversation

@anvacaru
Copy link
Copy Markdown
Contributor

@anvacaru anvacaru commented Apr 16, 2025

Adding support for the new execution-spec-tests conformance suite of the EVM.
Creating a new dir under tests/execution-spec-tests with:

  1. a new get-execution-spec-tests.sh script inspired by the one provided in the documentation (source) that downloads a given version of the JSON fixtures from the official GH release page.
  2. a new kevm-pyk/src/tests/integration/test_execution_spec_tests.py pytest file that executes four categories of tests: blockchain, blockchain-engine, state and transactions. Duplicated logic from the old kevm-pyk/src/tests/integration/test_conformance.py has been refactored under kevm-pyk/src/tests/utils.py
  3. new execution-spec-tests/failing.llvm file.
  4. I added the test-fixtures and fixtures-failing-list Makefile rules that download the JSON fixtures. They are used on CI.

@anvacaru anvacaru marked this pull request as ready for review April 16, 2025 14:54
Comment thread tests/execution-spec-tests/get_execution_spec_tests.sh
Comment thread tests/execution-spec-tests/slow.llvm Outdated
@ehildenb
Copy link
Copy Markdown
Member

So now the test-integration will run both the old tests in the old format, and the new ones? And I guess the plan is that eventually we'll pass all the new ones and phase out the old ones?

Comment thread Makefile Outdated
Comment thread tests/execution-spec-tests/get_execution_spec_tests.sh
Comment thread tests/execution-spec-tests/get_execution_spec_tests.sh Outdated
@anvacaru anvacaru requested a review from ehildenb April 17, 2025 14:14
Comment thread Makefile Outdated
Copy link
Copy Markdown
Member

@ehildenb ehildenb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Co-authored-by: Everett Hildenbrandt <everett.hildenbrandt@gmail.com>
@automergerpr-permission-manager automergerpr-permission-manager Bot merged commit b3de342 into master Apr 17, 2025
12 checks passed
@automergerpr-permission-manager automergerpr-permission-manager Bot deleted the conformance-tests-update branch April 17, 2025 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants