feat(test-cli): add --extract-opcode-count opcode tracing (#3124)

* feat(fill-stateful): add --extract-opcode-count opcode tracing

Trace each execution-phase block via debug_traceBlockByHash with a JS
opcode-counting tracer and record per-opcode execution counts in the
fixture's _info.metadata.opcode_count, matching the standard fill
benchmark path.

- DebugRPC.trace_block_by_hash wraps debug_traceBlockByHash.
- ClientBackend gains debug_rpc + extract_opcode_count; a JS tracer
  constant; extract_block_opcode_count() aggregating per-tx opcode
  counts; reset_opcode_count() now zeroes the per-test tally.
- make_stateful_fixture traces execution-phase blocks only (setup
  blocks skipped) and accumulates onto the backend, which the filler
  already serializes into _info.metadata.opcode_count.
- fill_stateful plugin: --extract-opcode-count flag, debug_rpc wired
  into the backend, per-test reset_opcode_count in the t8n override.

* fix(fill-stateful): harden opcode extraction; decouple from verification

Address review of the --extract-opcode-count feature:

- Decouple metadata from benchmark verification: only accumulate onto
  the metadata opcode_count, never set benchmark_opcode_count. The
  live-client trace is recorded for analysis; it no longer activates
  target-opcode verification (which stays a t8n-path concern and could
  otherwise fail benchmark tests on >5% trace divergence).
- Make extract_block_opcode_count non-fatal: a debug_traceBlockByHash
  RPC error is logged and skipped instead of aborting the fill.
- Normalize opcode names: geth emits 'opcode 0xNN not defined' for
  undefined opcodes, which OpcodeCount could not parse and would raise
  on. Reduce such names to the bare 0xNN (UndefinedOpcode); drop any
  still-unrecognized name with a warning.

* feat(fill-stateful): verify opcode counts against declared targets

With --extract-opcode-count on, feed the live-client traced count to
benchmark opcode verification: a test declaring fixed/expected_opcode_
count now fails the fill when its live-client count diverges >5% from
the target. Stays None (verification skipped) when the flag is off.

* feat(fill-stateful): opcode counting across clients (normalize names + struct-log fallback)

- Normalize tracer opcode names case-insensitively so clients emitting
  non-canonical names (nethermind returns calldatasize / pusH1) are counted
  instead of dropped as unrecognized.
- Fall back to the universally-supported struct-log tracer
  (disableStack/Memory/Storage, counting structLogs[].op) when a client's JS
  tracer is unavailable — erroring or empty — so besu (no JS tracer) is
  covered. Cached per client to avoid re-probing the JS tracer every block.
- Update the --extract-opcode-count help text accordingly.

Verified against geth (JS), nethermind (JS + normalize) and besu (struct-log
fallback): all three produce identical opcode counts.

* feat(fill-stateful): record per-payload opcode counts as opcode_counts array

Replace the single aggregated _info.metadata.opcode_count with
_info.metadata.opcode_counts — an array with one entry per
engineNewPayloads block (opcode_counts[i] is the count for
engineNewPayloads[i], or null if its trace was unavailable), so
multi-block benchmarks keep per-payload granularity.

The list is emitted via FillResult.metadata (only when
--extract-opcode-count is on); reset_opcode_count is now a no-op so the
filler no longer writes the singular opcode_count for stateful fixtures.

* refactor: add fixture and reduce comment

* refactor: opcode count trace logic

* refactor: error handling for rpc call

---------

Co-authored-by: LouisTsai <q1030176@gmail.com>

Author: skylenet

docs/filling_tests/fill_stateful.md

@@ -13,7 +13,7 @@ The target client must expose:
 
 - `testing` (`testing_buildBlockV1`) — block construction with explicit transaction ordering.
 - `engine` — `engine_newPayloadVX`, `engine_forkchoiceUpdatedVX`.
-- `eth`, `debug` — chain queries and `debug_setHead` (or `debug_resetHead` on clients like Nethermind that lack `debug_setHead`) for between-test rewind.
+- `eth`, `debug` — chain queries and `debug_setHead` (or `debug_resetHead` on clients like Nethermind that lack `debug_setHead`) for between-test rewind. `--extract-opcode-count` additionally needs `debug_traceBlockByHash` with JS tracer support.
 - `web3` (optional) — `web3_clientVersion` is recorded into the fixture's `_info.filling-transition-tool` for traceability.
 
 The production-ready filler is `ethpandaops/geth:master`.
@@ -122,6 +122,7 @@ Optional:
 - `--default-{gas-price,max-fee-per-gas,max-priority-fee-per-gas,max-fee-per-blob-gas}` — pin per-session fees; defaults bump live-query values by `1.5×`.
 - `--output PATH` — default `./fixtures`.
 - `--clean` — wipe the output dir before filling.
+- `--extract-opcode-count` — after building each block, trace it via `debug_traceBlockByHash` (a JS opcode-counting tracer) and record per-opcode execution counts (execution-phase blocks only) in the fixture's `_info.metadata.opcode_counts` — an array with one entry per `engineNewPayloads` block (`opcode_counts[i]` is the count for `engineNewPayloads[i]`, or `null` if its trace was unavailable), so multi-block benchmarks keep per-payload granularity. When a benchmark test declares a target opcode count (`fixed_opcode_count`/`expected_opcode_count`), the live-client count is verified against it and the fill fails on >5% divergence. Requires the `debug` namespace with JS tracer support. Adds a full re-execution trace per block, so it is slow and opt-in.
 
 ## Output layout
 
@@ -135,7 +136,7 @@ Optional:
             └── <test>.json         # per-test setup + execution payloads
 ```
 
-Each `pre_run/<start_block_hash>.json` (a `StatefulPreRunFixture`) is replayed once per `benchmarkoor` run. Per-test fixtures (`BlockchainEngineStatefulFixture`) reference their setup file by hash: a fixture with `startBlockHash = 0xabc...` is preceded by `pre_run/0xabc....json`. Each per-test fixture carries `snapshotBlockNumber`/`Hash`, `startBlockNumber`/`Hash`, `setupEngineNewPayloads`, `engineNewPayloads`, plus a `benchmarkGasUsed` field and the EL build in `_info.filling-transition-tool`. The hash-based filename leaves room for multiple pre-run files (e.g. different setup variants off one snapshot) without coordinating names.
+Each `pre_run/<start_block_hash>.json` (a `StatefulPreRunFixture`) is replayed once per `benchmarkoor` run. Per-test fixtures (`BlockchainEngineStatefulFixture`) reference their setup file by hash: a fixture with `startBlockHash = 0xabc...` is preceded by `pre_run/0xabc....json`. Each per-test fixture carries `snapshotBlockNumber`/`Hash`, `startBlockNumber`/`Hash`, `setupEngineNewPayloads`, `engineNewPayloads`, plus a `benchmarkGasUsed` field and the EL build in `_info.filling-transition-tool`. With `--extract-opcode-count`, it also carries `_info.metadata.opcode_counts` (an array of per-opcode execution counts, one entry per `engineNewPayloads` block). The hash-based filename leaves room for multiple pre-run files (e.g. different setup variants off one snapshot) without coordinating names.
 
 !!! warning "Snapshot anchoring"
     `--snapshot-block` accepts a hash on purpose. Anchoring to `latest` works against a quiescent client, but a live reorg between session start and fixture write would silently re-anchor the fixture to a different block. The hash form rejects that.

packages/testing/src/execution_testing/cli/pytest_commands/plugins/fill_stateful/fill_stateful.py

@@ -132,6 +132,21 @@ def pytest_addoption(parser: pytest.Parser) -> None:
             "by hash. The produced fixtures are anchored by hash regardless."
         ),
     )
+    group.addoption(
+        "--extract-opcode-count",
+        action="store_true",
+        dest="extract_opcode_count",
+        default=False,
+        help=(
+            "Trace each built block via debug_traceBlockByHash and record "
+            "per-opcode execution counts in the fixture's "
+            "_info.metadata.opcode_counts. Uses a client-side JS tracer where "
+            "supported (geth/nethermind/erigon/reth) and falls back to the "
+            "struct-log tracer otherwise (besu). Requires the `debug` "
+            "namespace. Adds a full re-execution trace per block — slow; "
+            "opt-in."
+        ),
+    )
 
 
 def _resolve_session_fork(
@@ -475,9 +490,17 @@ def debug_rpc(eth_rpc: EthRPC) -> DebugRPC:
     return DebugRPC(eth_rpc.url)
 
 
+@pytest.fixture(scope="session")
+def extract_opcode_count(request: pytest.FixtureRequest) -> bool:
+    """Whether --extract-opcode-count block tracing is enabled."""
+    return request.config.getoption("extract_opcode_count")
+
+
 @pytest.fixture(scope="session")
 def client_backend(
     eth_rpc: ChainBuilderEthRPC,
+    debug_rpc: DebugRPC,
+    extract_opcode_count: bool,
     session_fork: Fork | TransitionFork,
     default_gas_price: int | None,
     default_max_fee_per_gas: int | None,
@@ -501,6 +524,8 @@ def client_backend(
         engine_rpc=eth_rpc.engine_rpc,
         eth_rpc=eth_rpc,
         fork=session_fork,
+        debug_rpc=debug_rpc,
+        extract_opcode_count=extract_opcode_count,
     )
 
     priority_fee = default_max_priority_fee_per_gas
@@ -719,7 +744,8 @@ def session_t8n(
 def t8n(
     session_t8n: ClientBackend,
 ) -> Generator[ClientBackend, None, None]:
-    """Override: no per-test reset needed for ClientBackend."""
+    """Override: zero per-test opcode counts (no-op unless enabled)."""
+    session_t8n.reset_opcode_count()
     yield session_t8n
 
 

packages/testing/src/execution_testing/client_clis/client_backend.py

@@ -11,16 +11,25 @@
 info, block boundaries, and pre-alloc declarations flow unchanged.
 """
 
+import re
 from pathlib import Path
 from typing import Any, ClassVar, Dict, List, Optional
 
 from execution_testing.base_types import Bytes, Hash
 from execution_testing.exceptions import ExceptionBase, ExceptionMapper
 from execution_testing.forks import Fork, TransitionFork
-from execution_testing.rpc import EngineRPC, EthRPC, TestingRPC, Web3RPC
+from execution_testing.logging import get_logger
+from execution_testing.rpc import (
+    DebugRPC,
+    EngineRPC,
+    EthRPC,
+    TestingRPC,
+    Web3RPC,
+)
 from execution_testing.rpc.rpc_types import (
     ForkchoiceState,
     GetPayloadResponse,
+    JSONRPCError,
     PayloadAttributes,
     PayloadStatusEnum,
 )
@@ -35,9 +44,91 @@
     Result,
     Traces,
     TransitionToolOutput,
+    validate_opcode,
 )
 from .transition_tool import TransitionTool
 
+logger = get_logger(__name__)
+
+# Per-tx opcode tally; clients ship no built-in opcode-count tracer.
+OPCODE_COUNT_TRACER_JS = """{
+    counts: {},
+    step: function(log) {
+        var op = log.op.toString();
+        this.counts[op] = (this.counts[op] || 0) + 1;
+    },
+    fault: function() {},
+    result: function() { return this.counts; }
+}"""
+
+# Keep each struct-log step small.
+STRUCT_LOG_TRACER_CONFIG = {
+    "disableStack": True,
+    "disableMemory": True,
+    "disableStorage": True,
+}
+
+
+def _normalize_opcode_name(name: str) -> str | None:
+    """
+    Map a tracer opcode name to one ``OpcodeCount`` accepts.
+
+    Handles non-canonical casing (nethermind) and geth's
+    ``"opcode 0xNN not defined"``; unrecognized names are dropped.
+    """
+    for candidate in (name, name.upper()):
+        try:
+            validate_opcode(candidate)
+            return candidate
+        except Exception:
+            continue
+    match = re.search(r"0x[0-9a-fA-F]+", name)
+    if match is not None:
+        return match.group(0)
+    logger.warning(f"opcode trace: dropping unrecognized {name!r}")
+    return None
+
+
+def _opcode_count_from_js_tracer(traces: Any) -> OpcodeCount:
+    """Aggregate the per-tx ``{opcode: count}`` maps the JS tracer emits."""
+    counts: Dict[str, int] = {}
+    for entry in traces or []:
+        if not isinstance(entry, dict):
+            continue
+        tx_counts = entry.get("result")
+        # Clients that ignore the JS tracer echo struct logs.
+        if not isinstance(tx_counts, dict) or "structLogs" in tx_counts:
+            continue
+        for opcode, count in tx_counts.items():
+            key = _normalize_opcode_name(opcode)
+            if key is None or not isinstance(count, int):
+                continue
+            counts[key] = counts.get(key, 0) + count
+    return OpcodeCount.model_validate(counts)
+
+
+def _opcode_count_from_struct_logs(traces: Any) -> OpcodeCount:
+    """Count ``structLogs[].op`` entries, one per executed opcode."""
+    counts: Dict[str, int] = {}
+    for entry in traces or []:
+        if not isinstance(entry, dict):
+            continue
+        # Some clients return the struct-log result unwrapped.
+        result = entry.get("result")
+        if not isinstance(result, dict):
+            result = entry
+        for step in result.get("structLogs") or []:
+            if not isinstance(step, dict):
+                continue
+            op = step.get("op")
+            if not op:
+                continue
+            key = _normalize_opcode_name(op)
+            if key is None:
+                continue
+            counts[key] = counts.get(key, 0) + 1
+    return OpcodeCount.model_validate(counts)
+
 
 class ClientBackendExceptionMapper(ExceptionMapper):
     """
@@ -89,12 +180,18 @@ def __init__(
         engine_rpc: EngineRPC,
         eth_rpc: EthRPC,
         fork: Fork | TransitionFork,
+        debug_rpc: DebugRPC | None = None,
+        extract_opcode_count: bool = False,
     ) -> None:
         """Initialize with the RPC clients and the session fork."""
         self.testing_rpc = testing_rpc
         self.engine_rpc = engine_rpc
         self.eth_rpc = eth_rpc
         self.fork = fork
+        self.debug_rpc = debug_rpc
+        self.extract_opcode_count = extract_opcode_count
+        # Sticky fallback to struct logs (besu has no JS tracer).
+        self._js_tracer_unsupported = False
         self.exception_mapper = ClientBackendExceptionMapper()
         self.snapshot_block = None
         self.start_block = None
@@ -126,7 +223,7 @@ def reset_traces(self) -> None:
         return
 
     def reset_opcode_count(self) -> None:
-        """No-op — opcode counting not supported."""
+        """No-op — ``opcode_count`` stays ``None`` so the filler skips it."""
         return
 
     def set_cache(self, *, key: str) -> bool:
@@ -209,6 +306,45 @@ def evaluate(
             ),
         )
 
+    def extract_block_opcode_count(
+        self, block_hash: Hash
+    ) -> OpcodeCount | None:
+        """
+        Tally executed opcodes for a block via ``debug_traceBlockByHash``.
+
+        ``None`` when ``--extract-opcode-count`` is off or the trace
+        fails (logged, never fatal). Prefers the JS tracer; a client
+        that rejects it falls back to struct logs for the session,
+        while transient errors only skip the block.
+        """
+        if not self.extract_opcode_count or self.debug_rpc is None:
+            return None
+
+        try:
+            if not self._js_tracer_unsupported:
+                try:
+                    traces = self._trace_block(
+                        block_hash, {"tracer": OPCODE_COUNT_TRACER_JS}
+                    )
+                    return _opcode_count_from_js_tracer(traces)
+                except JSONRPCError as e:
+                    logger.info(f"JS tracer rejected ({e}); using struct logs")
+                    self._js_tracer_unsupported = True
+            traces = self._trace_block(block_hash, STRUCT_LOG_TRACER_CONFIG)
+            return _opcode_count_from_struct_logs(traces)
+        except Exception as e:
+            logger.warning(f"opcode trace failed for block {block_hash}: {e}")
+            return None
+
+    def _trace_block(
+        self, block_hash: Hash, tracer_config: Dict[str, Any]
+    ) -> Any:
+        """Raw ``debug_traceBlockByHash`` call; exceptions propagate."""
+        assert self.debug_rpc is not None
+        return self.debug_rpc.trace_block_by_hash(
+            str(block_hash), tracer_config
+        )
+
     def _payload_attributes(
         self,
         env: Any,

packages/testing/src/execution_testing/rpc/rpc.py

@@ -1301,6 +1301,15 @@ def trace_call(self, tr: dict[str, str], block_number: str) -> Any | None:
             request=RPCCall(method="traceCall", params=params)
         ).result_or_raise()
 
+    def trace_block_by_hash(
+        self, block_hash: str, tracer_config: dict[str, Any]
+    ) -> Any:
+        """`debug_traceBlockByHash`: Trace every transaction in a block."""
+        params = [block_hash, tracer_config]
+        return self.post_request(
+            request=RPCCall(method="traceBlockByHash", params=params)
+        ).result_or_raise()
+
     def set_head(self, block_number: str) -> None:
         """`debug_setHead`: Reset chain head to the given block number."""
         self.post_request(

packages/testing/src/execution_testing/specs/blockchain.py

@@ -1465,6 +1465,8 @@ def make_stateful_fixture(
 
         setup_payloads: List[FixtureEngineNewPayload] = []
         execution_payloads: List[FixtureEngineNewPayload] = []
+        # Aligned 1:1 with execution_payloads; None when no trace.
+        execution_opcode_counts: List[Dict[str, int] | None] = []
         head_hash = start_block_hash
         benchmark_gas_used: int | None = None
         benchmark_opcode_count: OpcodeCount | None = None
@@ -1485,21 +1487,29 @@ def make_stateful_fixture(
             payload = payload_metadata_to_fixture(
                 built_block.engine_payload, phase=block.phase
             )
+            # The client's authoritative block hash (the FixtureHeader RLP
+            # hash diverges — the client picks fields like gas_limit).
+            client_hash = Hash(
+                built_block.engine_payload.payload_response.execution_payload.block_hash
+            )
             if payload.phase == TestPhase.SETUP:
                 setup_payloads.append(payload)
             else:
                 execution_payloads.append(payload)
+                block_opcode_count = t8n.extract_block_opcode_count(
+                    client_hash
+                )
+                execution_opcode_counts.append(
+                    block_opcode_count.model_dump()
+                    if block_opcode_count is not None
+                    else None
+                )
                 if self.operation_mode == OpMode.BENCHMARKING:
                     benchmark_gas_used = int(built_block.result.gas_used)
-                    benchmark_opcode_count = built_block.result.opcode_count
-            # Overwrite the block_hash apply_new_parent just recorded —
-            # it's the FixtureHeader-recomputed RLP hash, which diverges
-            # from the client's authoritative hash (client picks fields
-            # like gas_limit). The next block's parent_hash must point at
-            # what the client actually built.
-            client_hash = Hash(
-                built_block.engine_payload.payload_response.execution_payload.block_hash
-            )
+                    # Consumed by BenchmarkTest's opcode-count verification.
+                    benchmark_opcode_count = block_opcode_count
+            # apply_new_parent records the RLP hash; the next block's
+            # parent_hash must point at what the client actually built.
             env = apply_new_parent(built_block.env, built_block.header)
             env = env.copy(
                 block_hashes={
@@ -1525,11 +1535,15 @@ def make_stateful_fixture(
                 else None
             ),
         )
+        metadata: Dict[str, Any] = {}
+        if t8n.extract_opcode_count:
+            metadata["opcode_counts"] = execution_opcode_counts
         return FillResult(
             fixture=fixture,
             gas_optimization=None,
             benchmark_gas_used=benchmark_gas_used,
             benchmark_opcode_count=benchmark_opcode_count,
+            metadata=metadata,
             post_verifications=PostVerifications.from_alloc(self.post),
         )