Skip to content

Add per-opcode weight profiler to benchmark reports#302

Open
mokita-j wants to merge 18 commits into
mainfrom
feature/add-weight-report
Open

Add per-opcode weight profiler to benchmark reports#302
mokita-j wants to merge 18 commits into
mainfrom
feature/add-weight-report

Conversation

@mokita-j

@mokita-j mokita-j commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in opcode-level profiler to retester benchmark. Samples K txs per step path (first / middle / last), traces each via pallet-revive's ExecutionTracer, and emits per-opcode ref_time + proof_size + counts into report.json. The HTML grows an Opcode Profile section with per-step min/mean/max band charts, sortable per-platform tables, and
cross-platform category bars.

Sampling the same step path at multiple lifecycle points allows us to analyze future optimizations like hot/cold storage pricing.

Depends on PR#12374 (adds state_callRecorded). Without it, trace replay over-accounts proof_size and tail extrinsics hitCheckWeight::ExhaustsResources.

Features

  • CLI: --benchmark.profile-watched-txs master switch + --benchmark.profile-samples-per-step-path (default 3), --benchmark.profile-concurrency, --benchmark.profile-step-limit, --benchmark.profile-all
  • Run Benchmarks workflow (.github/workflows/benchmark.yaml): four new dispatch inputs — profile-watched-txs (toggle, default off), profile-samples-per-step-path, profile-concurrency, profile-step-limit. When the toggle is on, the polkadot-sdk build steps add --cfg revive_debug to both RUSTFLAGS and WASM_BUILD_RUSTFLAGS
  • report.json per platform × mode × workload: workload-aggregate opcodes (top-N=64 + "Other" overflow), per-tx breakdown (tx_profiles[]), weight_consumed_* / base_call_weight_* / unattributed residual, embedded opcode_catalog (byte → name + category).
  • HTML Opcode Profile section: per-step ref_time/proof_size band charts, per-platform stacked-by-opcode bars, sortable opcode tables with inline share %, cross-platform category bar charts (with a VM-exclusive row for one-sided categories).
  • Constraint: profiling requires network = westend. Only the asset-hub-westend-runtime defines DebugEnabled = ConstBool<{ cfg!(revive_debug) }>; asset-hub-polkadot-runtime (from polkadot-fellows/runtimes) has no equivalent gate, so the cfg would be a silent no-op there. A validate-inputs job at the top of the workflow fails fast on the bad combination so we don't burn CI compiling polkadot-sdk for a run that can't produce traces.

Design decisions

  1. K-sample first/middle/last per step path, not random and not every tx. Same operation at cold / steady / end-of-workload makes the band charts useful for downstream pricing work; also bounds trace_tx cost independently of total tx count.
  2. Trace per-tx (trace_tx), not per-block. trace_block's response size scales with #extrinsics × per-tx step count and overruns both the WASM heap (128 MB) and subxt's 10 MB JSON-RPC frame on EVM-heavy workloads. Per-tx bounds the response to one tx's trace.
  3. Use state_callRecorded, not subxt's runtime API. Plain state_call doesn't install ProofSizeExt → reclaim breaks → replay diverges from authoring. We call the recorded variant via raw RPC.

Bumped pallet-revive 0.11.3 → 0.18.0 (the version that exposes ExecutionStepKind, needed for the PVM syscall name lookup).

@socket-security

socket-security Bot commented Jun 16, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedsp-runtime@​47.0.08210093100100
Updatedpallet-revive@​0.11.3 ⏵ 0.18.092 -7100100100100
Addedsp-core@​41.0.010010093100100
Addedstrum@​0.26.310010093100100

View full report

@elle-j elle-j mentioned this pull request Jun 16, 2026
@mokita-j

Copy link
Copy Markdown
Contributor Author

Report with the weight profiler can be consulted here: https://paritytech.github.io/revive-differential-tests/reports/benchmark-2026-06-16-160848.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant