refactor(specs): port state refactor to older forks (#2750)

* refactor(specs): apply state refactor to bpo forks

* refactor(specs): apply state refactor to post-merge forks

Apply the state tracker refactor (originally done for amsterdam) to the
remaining post-merge forks: osaka, prague, cancun, shanghai, and paris.

For each fork:
- Delete fork-specific state.py and trie.py
- Add fork-specific state_tracker.py with BlockState, TransactionState,
  and the associated read/write helpers
- Update fork.py to wrap chain.state in a BlockState, thread
  TransactionState through check_transaction / process_transaction /
  system transactions / withdrawals, and apply the block diff via
  apply_changes_to_state at the end of state_transition
- Update vm/__init__.py to use BlockState/TransactionState and import
  Trie from ethereum.merkle_patricia_trie
- Update interpreter to use copy_tx_state/restore_tx_state in place of
  begin/commit/rollback_transaction
- Update utils/message.py, vm/eoa_delegation.py, and vm/instructions/*
  to read state via evm.message.tx_env.state
- Update blocks.py docstring cross-refs to the shared state / trie
  modules

* refactor(specs): apply state refactor to pre-merge forks

Apply the state tracker refactor (originally done for amsterdam) to the
remaining pre-merge forks: gray_glacier, arrow_glacier, london, berlin,
muir_glacier, istanbul, constantinople, byzantium, spurious_dragon,
tangerine_whistle, dao_fork, homestead, and frontier.

For each fork:
- Delete fork-specific state.py and trie.py
- Add fork-specific state_tracker.py with BlockState, TransactionState,
  and the associated read/write helpers
- Update fork.py to wrap chain.state in a BlockState, thread
  TransactionState through check_transaction / process_transaction /
  block-reward payout, and apply the block diff via
  apply_changes_to_state at the end of state_transition
- Update vm/__init__.py to use BlockState/TransactionState and import
  Trie from ethereum.merkle_patricia_trie
- Update interpreter to use copy_tx_state/restore_tx_state in place of
  begin/commit/rollback_transaction
- Update utils/message.py and vm/instructions/* to read state via
  evm.message.tx_env.state
- Update blocks.py docstring cross-refs to the shared state / trie
  modules

For pre-merge forks, the state_tracker also exposes `create_ether`
(used for paying ommer/miner rewards) and the pre-EIP-161 forks have
`destroy_touched_empty_accounts` and `touch_account`. Pre-byzantium
receipts include an intermediate state root, computed by committing the
tx's writes to the block state and calling
`pre_state.compute_state_root_and_trie_changes` on the accumulated
block diff.

The DAO fork irregular state transition now creates a throwaway
BlockState/TransactionState, applies the moves, and flushes the diff
back to `chain.state`.

* fix(specs): track storage clears across pre-EIP-6780 destructions

Pre-EIP-6780 ``SELFDESTRUCT`` fully wipes an account's storage, and a
later ``CREATE2`` at the same address must start from empty storage.
After the state refactor, ``destroy_storage`` only removed tx-level
writes, so reads still fell through to the pre_state's storage trie —
``CREATE2`` then saw ``account_has_storage`` return True and bailed
without bumping the nonce.

Track wiped addresses in a new ``storage_clears`` set on ``BlockState``,
``TransactionState``, and ``BlockDiff``. Reads short-circuit on it, and
``compute_state_root_and_trie_changes`` / ``apply_changes_to_state``
drop the corresponding storage tries before re-applying writes, so
post-wipe writes begin from empty storage.

Applies to Frontier through Shanghai; post-6780 forks leave the set
empty.

* refactor(evm_tools): wire t8n through ForkLoad accessors

The t8n tool kept a mix of direct imports from ``amsterdam`` and raw
``_module(...)`` lookups into each fork's ``state_tracker`` and
``block_access_lists`` modules, and ``ForkLoad`` still carried fallback
shims for helpers (``State``, ``set_account``, ``copy_trie``, ``root``,
etc.) that now live exclusively in the shared ``ethereum.state`` and
``ethereum.merkle_patricia_trie`` modules.

Collapse all three:

* Drop the fork-independent shims from ``ForkLoad`` and import the
  helpers directly at the t8n / fixture-loader call sites.
* Expose ``BlockState``, ``TransactionState``, ``extract_block_diff``,
  ``incorporate_tx_into_block``, ``BlockAccessIndex``,
  ``BlockAccessListBuilder``, and ``validate_block_access_list_gas_limit``
  as ``ForkLoad`` properties so t8n never has to hardcode ``amsterdam``
  or reach into ``_module``.
* Fix ``pay_block_rewards`` to wrap ``BlockState`` in a
  ``TransactionState`` via the fork's own state_tracker (the previous
  code passed the block-scoped state straight to ``create_ether``,
  which expects the tx-scoped wrapper).
* Remove the now-dead ``has_block_state`` feature flag and its
  fallback branches — every fork has a ``state_tracker`` module.

* chore(tests): adjust json_loader tests for the state refactor

Point the json_loader tests at the shared ``ethereum.state`` and
``ethereum.merkle_patricia_trie`` modules now that per-fork ``state``
and ``trie`` modules no longer exist, and switch to the State-method
accessors (``state.get_storage(...)`` etc.) instead of the old
module-level functions.

Skip ``test_optimized_state`` wholesale: the optimized state integration
still imports per-fork ``state`` modules and exposes a pre-refactor
``destroy_storage(state, addr)`` API, both removed by the refactor.
Tracked by #2256
pending the optimized state redesign.

Extend ``vulture_whitelist.py`` to cover the ``@add_item``-registered
``begin_transaction`` / ``commit_transaction`` / ``rollback_transaction``
patches in ``ethereum_optimized.state_db``.

* refactor(specs): drop fork_types re-exports of shared state types

The state refactor initially re-imported ``Account``, ``Address``,
``EMPTY_ACCOUNT``, and ``EMPTY_CODE_HASH`` from ``ethereum.state`` into
each pre-merge fork's ``fork_types.py`` (plus a matching ``__all__``)
so existing call sites like ``from .fork_types import Address`` could
stay untouched. ``Root`` was similarly duplicated as ``Root = Hash32``
in every fork.

``docc`` does not follow ``__all__`` re-exports when resolving
cross-references, which breaks ``just docs-spec`` on the first consumer
that imports ``EMPTY_ACCOUNT`` through a fork's ``fork_types``.

Drop the re-exports from all 13 pre-merge forks and import the shared
symbols directly from ``ethereum.state`` at every call site — matching
the shape that shanghai / paris already use. Each ``fork_types.py``
collapses to ``Bloom``, ``encode_account``, and the ``Account`` type
import that ``encode_account`` needs.

Also update ``tests/json_loader/test_optimized_state.py`` to import
``EMPTY_ACCOUNT`` from ``ethereum.state`` rather than through the
dropped ``frontier.fork_types`` re-export.

* refactor(specs): address PR #2750 review feedback

Apply review suggestions from @SamWilsn:

* ``ethereum.state.BlockDiff.storage_clears`` docstring switches to
  markdown style with a cross-ref to ``storage_changes``, so docc
  renders the link.

* In every fork's ``state_tracker.get_account``, replace the
  ``isinstance(account, Account)`` guard with an explicit
  ``if account is None`` check. The ``isinstance`` was a leftover from
  when ``fork_types.Account`` was a separate class; now that every
  fork pulls ``Account`` directly from ``ethereum.state``, the
  ``is None`` form expresses the intent ("is there an account?")
  rather than the implementation ("is this an account?").

* Reintroduce the ``create_ether`` helper in every post-merge fork's
  state tracker (paris through bpo5). It was dropped during the
  post-merge state refactor because the PoS forks no longer
  ``pay_rewards`` to a miner, but it remains the natural helper for
  ``process_withdrawals``. Updates the withdrawal call sites in
  shanghai onwards (paris itself has no caller; ``create_ether`` is
  added there purely to keep sibling parity with london and shanghai).
  Also relocate ``create_ether`` in every pre-merge fork from its
  legacy spot in the trailing "Pre-EIP-161 cleanup" section to
  immediately after ``move_ether`` — they are structurally the same
  kind of balance mutator, and the new placement is consistent
  across every fork.

* Rewrite the ``untracked_state`` comment in
  ``process_checked_system_transaction`` (prague through bpo5) to
  make the throwaway-scope and "always calls
  process_unchecked_system_transaction below" contracts explicit.

Author: gurukamath

src/ethereum/forks/amsterdam/fork.py

@@ -71,6 +71,7 @@
     BlockState,
     TransactionState,
     account_exists_and_is_empty,
+    create_ether,
     destroy_account,
     extract_block_diff,
     get_account,
@@ -696,14 +697,14 @@ def process_checked_system_transaction(
         Output of processing the system transaction.
 
     """
-    # Read through BlockState (not pre-state) so that a system contract
-    # deployed by an earlier transaction in the same block is visible.
-    # See EIP-7002 and EIP-7251 for this edge case.
-    #
-    # This read is not recorded in the state tracker.
-    # However, this is fine because `process_unchecked_system_transaction`
-    # does its own get_account on the TransactionState that we do incorporate
-    # into BlockState.
+    # Pre-check that the system contract has code. We use a throwaway
+    # TransactionState here that is *never* propagated back to BlockState
+    # (no incorporate_tx_into_block call); the same get_account / get_code
+    # lookups are performed and properly tracked by
+    # process_unchecked_system_transaction below, which this function
+    # always calls. Reading via a TransactionState (rather than directly
+    # against pre_state) lets us see system contracts deployed earlier in
+    # the same block — see EIP-7002 and EIP-7251 for this edge case.
     untracked_state = TransactionState(parent=block_env.state)
     system_contract_code = get_code(
         untracked_state,
@@ -1108,9 +1109,7 @@ def process_withdrawals(
             rlp.encode(wd),
         )
 
-        current_balance = get_account(wd_state, wd.address).balance
-        new_balance = current_balance + wd.amount * GWEI_TO_WEI
-        set_account_balance(wd_state, wd.address, new_balance)
+        create_ether(wd_state, wd.address, wd.amount * GWEI_TO_WEI)
 
     incorporate_tx_into_block(wd_state, block_env.block_access_list_builder)
 

src/ethereum/forks/amsterdam/state_tracker.py

@@ -140,10 +140,10 @@ def get_account(tx_state: TransactionState, address: Address) -> Account:
 
     """
     account = get_account_optional(tx_state, address)
-    if isinstance(account, Account):
-        return account
-    else:
+    if account is None:
         return EMPTY_ACCOUNT
+    else:
+        return account
 
 
 def get_code(tx_state: TransactionState, code_hash: Hash32) -> Bytes:
@@ -568,6 +568,29 @@ def increase_recipient_balance(recipient: Account) -> None:
     modify_state(tx_state, recipient_address, increase_recipient_balance)
 
 
+def create_ether(
+    tx_state: TransactionState, address: Address, amount: U256
+) -> None:
+    """
+    Add newly created ether to an account.
+
+    Parameters
+    ----------
+    tx_state :
+        The transaction state.
+    address :
+        Address of the account to which ether is added.
+    amount :
+        The amount of ether to be added to the account of interest.
+
+    """
+
+    def increase_balance(account: Account) -> None:
+        account.balance += amount
+
+    modify_state(tx_state, address, increase_balance)
+
+
 def set_account_balance(
     tx_state: TransactionState, address: Address, amount: U256
 ) -> None:

src/ethereum/forks/arrow_glacier/blocks.py

@@ -18,8 +18,9 @@
 from ethereum_types.numeric import U256, Uint
 
 from ethereum.crypto.hash import Hash32
+from ethereum.state import Address, Root
 
-from .fork_types import Address, Bloom, Root
+from .fork_types import Bloom
 from .transactions import (
     AccessListTransaction,
     FeeMarketTransaction,
@@ -69,13 +70,14 @@ class Header:
     Root hash ([`keccak256`]) of the state trie after executing all
     transactions in this block. It represents the state of the Ethereum Virtual
     Machine (EVM) after all transactions in this block have been processed. It
-    is computed using the [`state_root()`] function, which computes the root
-    of the Merkle-Patricia [Trie] representing the Ethereum world state.
+    is computed using [`compute_state_root_and_trie_changes()`][changes],
+    which computes the root of the Merkle-Patricia [Trie] representing the
+    Ethereum world state after applying the block's state changes.
 
     [`keccak256`]: ref:ethereum.crypto.hash.keccak256
-    [`state_root()`]: ref:ethereum.forks.arrow_glacier.state.state_root
-    [Trie]: ref:ethereum.forks.arrow_glacier.trie.Trie
-    """
+    [changes]: ref:ethereum.state.State.compute_state_root_and_trie_changes
+    [Trie]: ref:ethereum.merkle_patricia_trie.Trie
+    """  # noqa: E501
 
     transactions_root: Root
     """
@@ -85,8 +87,8 @@ class Header:
     transactions as the parameter.
 
     [`keccak256`]: ref:ethereum.crypto.hash.keccak256
-    [`root()`]: ref:ethereum.forks.arrow_glacier.trie.root
-    [Trie]: ref:ethereum.forks.arrow_glacier.trie.Trie
+    [`root()`]: ref:ethereum.merkle_patricia_trie.root
+    [Trie]: ref:ethereum.merkle_patricia_trie.Trie
     """
 
     receipt_root: Root
@@ -96,8 +98,8 @@ class Header:
     function over the Merkle-Patricia [trie] constructed from the receipts.
 
     [`keccak256`]: ref:ethereum.crypto.hash.keccak256
-    [`root()`]: ref:ethereum.forks.arrow_glacier.trie.root
-    [Trie]: ref:ethereum.forks.arrow_glacier.trie.Trie
+    [`root()`]: ref:ethereum.merkle_patricia_trie.root
+    [Trie]: ref:ethereum.merkle_patricia_trie.Trie
     """
 
     bloom: Bloom

src/ethereum/forks/arrow_glacier/fork.py

@@ -28,6 +28,13 @@
     InvalidSenderError,
     NonceMismatchError,
 )
+from ethereum.merkle_patricia_trie import root, trie_set
+from ethereum.state import (
+    EMPTY_CODE_HASH,
+    Address,
+    State,
+    apply_changes_to_state,
+)
 
 from . import vm
 from .blocks import Block, Header, Log, Receipt, encode_receipt
@@ -36,17 +43,18 @@
     InsufficientMaxFeePerGasError,
     PriorityFeeGreaterThanMaxFeeError,
 )
-from .fork_types import EMPTY_CODE_HASH, Address
-from .state import (
-    State,
+from .state_tracker import (
+    BlockState,
+    TransactionState,
     account_exists_and_is_empty,
     create_ether,
     destroy_account,
     destroy_touched_empty_accounts,
+    extract_block_diff,
     get_account,
+    incorporate_tx_into_block,
     increment_nonce,
     set_account_balance,
-    state_root,
 )
 from .transactions import (
     AccessListTransaction,
@@ -59,7 +67,6 @@
     recover_sender,
     validate_transaction,
 )
-from .trie import root, trie_set
 from .utils.message import prepare_message
 from .vm.gas import GasCosts
 from .vm.interpreter import process_message_call
@@ -174,9 +181,11 @@ def state_transition(chain: BlockChain, block: Block) -> None:
     validate_header(chain, block.header)
     validate_ommers(block.ommers, block.header, chain)
 
+    block_state = BlockState(pre_state=chain.state)
+
     block_env = vm.BlockEnvironment(
         chain_id=chain.chain_id,
-        state=chain.state,
+        state=block_state,
         block_gas_limit=block.header.gas_limit,
         block_hashes=get_last_256_block_hashes(chain),
         coinbase=block.header.coinbase,
@@ -191,7 +200,12 @@ def state_transition(chain: BlockChain, block: Block) -> None:
         transactions=block.transactions,
         ommers=block.ommers,
     )
-    block_state_root = state_root(block_env.state)
+    block_diff = extract_block_diff(block_state)
+    block_state_root, _ = chain.state.compute_state_root_and_trie_changes(
+        block_diff.account_changes,
+        block_diff.storage_changes,
+        block_diff.storage_clears,
+    )
     transactions_root = root(block_output.transactions_trie)
     receipt_root = root(block_output.receipts_trie)
     block_logs_bloom = logs_bloom(block_output.block_logs)
@@ -209,6 +223,7 @@ def state_transition(chain: BlockChain, block: Block) -> None:
     if block_logs_bloom != block.header.bloom:
         raise InvalidBlock
 
+    apply_changes_to_state(chain.state, block_diff)
     chain.blocks.append(block)
     if len(chain.blocks) > 255:
         # Real clients have to store more blocks to deal with reorgs, but the
@@ -430,6 +445,7 @@ def check_transaction(
     block_env: vm.BlockEnvironment,
     block_output: vm.BlockOutput,
     tx: Transaction,
+    tx_state: TransactionState,
 ) -> Tuple[Address, Uint]:
     """
     Check if the transaction is includable in the block.
@@ -442,6 +458,8 @@ def check_transaction(
         The block output for the current block.
     tx :
         The transaction.
+    tx_state :
+        The transaction state tracker.
 
     Returns
     -------
@@ -472,7 +490,7 @@ def check_transaction(
     if tx.gas > gas_available:
         raise GasUsedExceedsLimitError("gas used exceeds limit")
     sender_address = recover_sender(block_env.chain_id, tx)
-    sender_account = get_account(block_env.state, sender_address)
+    sender_account = get_account(tx_state, sender_address)
 
     if isinstance(tx, FeeMarketTransaction):
         if tx.max_fee_per_gas < tx.max_priority_fee_per_gas:
@@ -581,7 +599,7 @@ def apply_body(
     for i, tx in enumerate(map(decode_transaction, transactions)):
         process_transaction(block_env, block_output, tx, Uint(i))
 
-    pay_rewards(block_env.state, block_env.number, block_env.coinbase, ommers)
+    pay_rewards(block_env, ommers)
 
     return block_output
 
@@ -663,9 +681,7 @@ def validate_ommers(
 
 
 def pay_rewards(
-    state: State,
-    block_number: Uint,
-    coinbase: Address,
+    block_env: vm.BlockEnvironment,
     ommers: Tuple[Header, ...],
 ) -> None:
     """
@@ -684,25 +700,24 @@ def pay_rewards(
 
     Parameters
     ----------
-    state :
-        Current account state.
-    block_number :
-        Position of the block within the chain.
-    coinbase :
-        Address of account which receives block reward and transaction fees.
+    block_env :
+        The block scoped environment.
     ommers :
         List of ommers mentioned in the current block.
 
     """
+    rewards_state = TransactionState(parent=block_env.state)
     ommer_count = U256(len(ommers))
     miner_reward = BLOCK_REWARD + (ommer_count * (BLOCK_REWARD // U256(32)))
-    create_ether(state, coinbase, miner_reward)
+    create_ether(rewards_state, block_env.coinbase, miner_reward)
 
     for ommer in ommers:
         # Ommer age with respect to the current block.
-        ommer_age = U256(block_number - ommer.number)
+        ommer_age = U256(block_env.number - ommer.number)
         ommer_miner_reward = ((U256(8) - ommer_age) * BLOCK_REWARD) // U256(8)
-        create_ether(state, ommer.coinbase, ommer_miner_reward)
+        create_ether(rewards_state, ommer.coinbase, ommer_miner_reward)
+
+    incorporate_tx_into_block(rewards_state)
 
 
 def process_transaction(
@@ -735,6 +750,8 @@ def process_transaction(
         Index of the transaction in the block.
 
     """
+    tx_state = TransactionState(parent=block_env.state)
+
     trie_set(
         block_output.transactions_trie,
         rlp.encode(index),
@@ -750,21 +767,20 @@ def process_transaction(
         block_env=block_env,
         block_output=block_output,
         tx=tx,
+        tx_state=tx_state,
     )
 
-    sender_account = get_account(block_env.state, sender)
+    sender_account = get_account(tx_state, sender)
 
     effective_gas_fee = tx.gas * effective_gas_price
 
     gas = tx.gas - intrinsic_gas
-    increment_nonce(block_env.state, sender)
+    increment_nonce(tx_state, sender)
 
     sender_balance_after_gas_fee = (
         Uint(sender_account.balance) - effective_gas_fee
     )
-    set_account_balance(
-        block_env.state, sender, U256(sender_balance_after_gas_fee)
-    )
+    set_account_balance(tx_state, sender, U256(sender_balance_after_gas_fee))
 
     access_list_addresses = set()
     access_list_storage_keys = set()
@@ -780,6 +796,7 @@ def process_transaction(
         gas=gas,
         access_list_addresses=access_list_addresses,
         access_list_storage_keys=access_list_storage_keys,
+        state=tx_state,
         index_in_block=index,
         tx_hash=get_transaction_hash(encode_transaction(tx)),
     )
@@ -801,28 +818,28 @@ def process_transaction(
     transaction_fee = tx_gas_used_after_refund * priority_fee_per_gas
 
     # refund gas
-    sender_balance_after_refund = get_account(
-        block_env.state, sender
-    ).balance + U256(gas_refund_amount)
-    set_account_balance(block_env.state, sender, sender_balance_after_refund)
+    sender_balance_after_refund = get_account(tx_state, sender).balance + U256(
+        gas_refund_amount
+    )
+    set_account_balance(tx_state, sender, sender_balance_after_refund)
 
     # transfer miner fees
     coinbase_balance_after_mining_fee = get_account(
-        block_env.state, block_env.coinbase
+        tx_state, block_env.coinbase
     ).balance + U256(transaction_fee)
     if coinbase_balance_after_mining_fee != 0:
         set_account_balance(
-            block_env.state,
+            tx_state,
             block_env.coinbase,
             coinbase_balance_after_mining_fee,
         )
-    elif account_exists_and_is_empty(block_env.state, block_env.coinbase):
-        destroy_account(block_env.state, block_env.coinbase)
+    elif account_exists_and_is_empty(tx_state, block_env.coinbase):
+        destroy_account(tx_state, block_env.coinbase)
 
     for address in tx_output.accounts_to_delete:
-        destroy_account(block_env.state, address)
+        destroy_account(tx_state, address)
 
-    destroy_touched_empty_accounts(block_env.state, tx_output.touched_accounts)
+    destroy_touched_empty_accounts(tx_state, tx_output.touched_accounts)
 
     block_output.block_gas_used += tx_gas_used_after_refund
 
@@ -841,6 +858,8 @@ def process_transaction(
 
     block_output.block_logs += tx_output.logs
 
+    incorporate_tx_into_block(tx_state)
+
 
 def check_gas_limit(gas_limit: Uint, parent_gas_limit: Uint) -> bool:
     """