Skip to content

feat(benchmarks): fix benchmark for warm SSTORE/SLOAD for Osaka#1915

Merged
LouisTsai-Csie merged 3 commits intoethereum:forks/amsterdamfrom
jsign:jsign-fix-storage-access-warm-osaka
Dec 17, 2025
Merged

feat(benchmarks): fix benchmark for warm SSTORE/SLOAD for Osaka#1915
LouisTsai-Csie merged 3 commits intoethereum:forks/amsterdamfrom
jsign:jsign-fix-storage-access-warm-osaka

Conversation

@jsign
Copy link
Copy Markdown
Collaborator

@jsign jsign commented Dec 12, 2025

🗒️ Description

This PR fixes the test_storage_access_warm benchmark to work in Osaka.

For 20M gas benchmark value:

  • test_storage.py::test_storage_access_warm[fork_Osaka-blockchain_test-SLOAD]: Osaka 310M cycles. Prague 311M cycles
  • test_storage.py::test_storage_access_warm[fork_Osaka-blockchain_test-SSTORE new value]: Osaka 908M cycles. Prague 909M cycles.
  • test_storage.py::test_storage_access_warm[fork_Osaka-blockchain_test-SSTORE same value]: Osaka 636M cycles. Prague 637M cycles.

Pretty brutal for only being 20M gas.

🔗 Related Issues or PRs

#1695

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx tox -e static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

Comment thread tests/benchmark/compute/instruction/test_storage.py
Comment thread tests/benchmark/compute/instruction/test_storage.py
Comment thread tests/benchmark/compute/instruction/test_storage.py
@jsign jsign marked this pull request as ready for review December 12, 2025 21:23
@jsign jsign requested a review from LouisTsai-Csie December 12, 2025 21:23
@jsign
Copy link
Copy Markdown
Collaborator Author

jsign commented Dec 12, 2025

cc @jochem-brouwer if you want to take a look

@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 12, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.61%. Comparing base (9e161b6) to head (383427c).
⚠️ Report is 7 commits behind head on forks/amsterdam.

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #1915   +/-   ##
================================================
  Coverage            82.61%   82.61%           
================================================
  Files                  417      417           
  Lines                26862    26862           
  Branches              2492     2492           
================================================
  Hits                 22193    22193           
  Misses                4230     4230           
  Partials               439      439           
Flag Coverage Δ
unittests 82.61% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Collaborator

@LouisTsai-Csie LouisTsai-Csie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR! Leave some comments about the usage of tx_gas_limit.

@pytest.fixture
def tx_gas_limit(fork: Fork, gas_benchmark_value: int) -> int:
"""Return the transaction gas limit cap."""
return fork.transaction_gas_limit_cap() or gas_benchmark_value

One question for this test (test_storage_access_warm), I wonder if we should take care of the OOG issue for this test? It seems these benchmark test transaction would OOG, and my understanding is that the storage would revert due to OOG, will this affect the intention of this test?

Since this test (test_storage_access_cold) do consider this scenario.

Comment thread tests/benchmark/compute/instruction/test_storage.py Outdated
Comment thread tests/benchmark/compute/instruction/test_storage.py Outdated
Comment thread tests/benchmark/compute/instruction/test_storage.py Outdated
@jsign
Copy link
Copy Markdown
Collaborator Author

jsign commented Dec 14, 2025

Leave some comments about the usage of tx_gas_limit.

Fixed! I'll also do this change in other open PRs.

One question for this test (test_storage_access_warm), I wonder if we should take care of the OOG issue for this test? It seems these benchmark test transaction would OOG, and my understanding is that the storage would revert due to OOG, will this affect the intention of this test?

Since this test (test_storage_access_cold) do consider this scenario.

I think the non-OOG case is of main relevance for this warm flavor.. In this test we exhaust the gas limit in a single storage slot on warm accesses, so the potential post-state-root calculation would only imply a single storage slot. For the cold case, I added OOG and non-OOG since at a minimum non-OOG is quite relevant, since we target as many storage slots as possible, so those are many writes that will triger post-state root calculation.

Copy link
Copy Markdown
Collaborator

@LouisTsai-Csie LouisTsai-Csie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have recently bumped our default branch from forks/osaka to forks/amsterdam, could you please help update your base branch please. Otherwise, i have no other requested changes! Really appreciate the help, thanks a lot

@LouisTsai-Csie LouisTsai-Csie added C-refactor Category: refactor A-test-benchmark Area: execution_testing.benchmark and tests/benchmark labels Dec 15, 2025
@jsign jsign force-pushed the jsign-fix-storage-access-warm-osaka branch from 27ee240 to 7f7358d Compare December 15, 2025 11:35
@jsign jsign changed the base branch from forks/osaka to forks/amsterdam December 15, 2025 11:41
@jsign
Copy link
Copy Markdown
Collaborator Author

jsign commented Dec 15, 2025

@LouisTsai-Csie, all done.

@LouisTsai-Csie
Copy link
Copy Markdown
Collaborator

Thanks a lot!! i will merge after forks/amsterdam recreated.

Copy link
Copy Markdown
Member

@jochem-brouwer jochem-brouwer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments!

Comment thread tests/benchmark/compute/instruction/test_storage.py
Comment thread tests/benchmark/compute/instruction/test_storage.py
Comment thread tests/benchmark/compute/instruction/test_storage.py
Comment thread tests/benchmark/compute/instruction/test_storage.py
@jsign jsign force-pushed the jsign-fix-storage-access-warm-osaka branch from 7f7358d to 383427c Compare December 15, 2025 19:52
@LouisTsai-Csie
Copy link
Copy Markdown
Collaborator

LGTM

@LouisTsai-Csie LouisTsai-Csie merged commit 3afd624 into ethereum:forks/amsterdam Dec 17, 2025
11 checks passed
Carsons-Eels pushed a commit to Carsons-Eels/execution-specs that referenced this pull request Jan 6, 2026
Carsons-Eels pushed a commit to Carsons-Eels/execution-specs that referenced this pull request Jan 6, 2026
CPerezz pushed a commit to CPerezz/execution-specs that referenced this pull request Feb 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-test-benchmark Area: execution_testing.benchmark and tests/benchmark C-refactor Category: refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants