Skip to content

Commit 173e7f2

Browse files
authored
feat: external opcode metadata for archive sources (#180)
## Summary - Adds `opcodes` and `opcodes_file` fields to the archive source config, allowing opcode count data to be provided via an external JSON file when not using EEST fixtures - Opcode data is written to `opcode_count` at the top level of each test in `summary.json`, and the UI opcode heatmap/test detail views now read from this unified field (with fallback to `eest.info.opcode_count` for backward compatibility) - Adds archive source documentation to `docs/configuration.md` and `config.example.yaml` which were previously missing ## Config example ```yaml source: archive: file: https://github.com/.../artifacts/6222084759 opcodes: "opcodes_tracing.json" # Optional: separate archive for the opcodes file opcodes_file: https://github.com/.../artifacts/6222074312 steps: test: - "perf-devnet-3/testing/*.txt" ``` ## Test plan - [x] `go build ./pkg/executor/... ./pkg/config/...` compiles - [x] `go test ./pkg/executor/... ./pkg/config/...` passes - [x] `golangci-lint run` passes on changed packages - [x] TypeScript type-checks with `npx tsc --noEmit` - [x] Manual: run with archive source + opcodes config, verify `summary.json` contains `opcode_count` per test - [x] Manual: verify UI opcode heatmap renders for non-EEST suite
1 parent 58f1170 commit 173e7f2

10 files changed

Lines changed: 390 additions & 25 deletions

File tree

config.example.yaml

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,27 @@ runner:
129129
# # cleanup:
130130
# # - "tests/cleanup/*.txt"
131131
#
132-
# # Option 3: EEST (Ethereum Execution Spec Tests) fixtures from GitHub releases.
132+
# # Option 3: Archive source (ZIP or tar.gz, local or remote URL).
133+
# # Downloads (if URL), extracts, and discovers tests from the archive contents.
134+
# # GitHub Actions artifact URLs are auto-converted to API download endpoints.
135+
# # archive:
136+
# # file: https://github.com/NethermindEth/gas-benchmarks/actions/runs/23847558369/artifacts/6222084759
137+
# # pre_run_steps:
138+
# # - "perf-devnet-3/gas-bump.txt"
139+
# # - "perf-devnet-3/funding.txt"
140+
# # steps:
141+
# # setup:
142+
# # - "perf-devnet-3/setup/*.txt"
143+
# # test:
144+
# # - "perf-devnet-3/testing/*.txt"
145+
# # # Optional: External opcode metadata file within the archive.
146+
# # # JSON mapping test names to opcode counts: {"test_name": {"OPCODE": count}}
147+
# # # opcodes: "opcodes_tracing.json"
148+
# # # Optional: Separate archive containing the opcodes file.
149+
# # # If not set, the opcodes file is searched in the main archive.
150+
# # # opcodes_file: https://github.com/.../artifacts/6222074312
151+
#
152+
# # Option 4: EEST (Ethereum Execution Spec Tests) fixtures from GitHub releases.
133153
# # Downloads fixtures directly from ethereum/execution-spec-tests releases.
134154
# # Genesis files are auto-resolved per client type from the release.
135155
# # eest_fixtures:
@@ -141,7 +161,7 @@ runner:
141161
# # # fixtures_url: https://example.com/fixtures_benchmark.tar.gz
142162
# # # genesis_url: https://example.com/benchmark_genesis.tar.gz
143163
#
144-
# # Option 3b: EEST fixtures from GitHub Actions artifacts.
164+
# # Option 4b: EEST fixtures from GitHub Actions artifacts.
145165
# # Alternative to releases - downloads from workflow run artifacts.
146166
# # Uses the gh CLI if available, otherwise falls back to the GitHub REST API
147167
# # (requires runner.github_token or BENCHMARKOOR_RUNNER_GITHUB_TOKEN).
@@ -154,7 +174,7 @@ runner:
154174
# # # genesis_artifact_run_id: "12345678901"
155175
# # # fixtures_subdir also works with artifacts
156176
#
157-
# # Option 3c: EEST fixtures from local directories.
177+
# # Option 4c: EEST fixtures from local directories.
158178
# # Points directly at already-extracted fixtures and genesis directories.
159179
# # No downloading or caching — useful for local development with locally-built EEST fixtures.
160180
# # eest_fixtures:
@@ -163,7 +183,7 @@ runner:
163183
# # # fixtures_subdir also works with local directories
164184
# # # fixtures_subdir: fixtures/blockchain_tests_engine_x # default
165185
#
166-
# # Option 3d: EEST fixtures from local .tar.gz tarballs.
186+
# # Option 4d: EEST fixtures from local .tar.gz tarballs.
167187
# # Extracts tarballs to a cache directory (keyed by content hash, re-extraction skipped
168188
# # if already cached). Useful when you have locally-built tarballs but haven't extracted them.
169189
# # eest_fixtures:

docs/configuration.md

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,7 +243,7 @@ runner:
243243

244244
#### Test Sources
245245

246-
Tests can be loaded from a local directory, a git repository, or EEST (Ethereum Execution Spec Tests) fixtures. Only one source type can be configured.
246+
Tests can be loaded from a local directory, a git repository, an archive file, or EEST (Ethereum Execution Spec Tests) fixtures. Only one source type can be configured.
247247

248248
##### Local Source
249249

@@ -299,6 +299,46 @@ tests:
299299
| `steps.test` | []string | No | Glob patterns for test phase files |
300300
| `steps.cleanup` | []string | No | Glob patterns for cleanup phase files |
301301

302+
##### Archive Source
303+
304+
Tests can be loaded from a ZIP or tar.gz archive file, either from a local path or a URL (including GitHub Actions artifacts).
305+
306+
```yaml
307+
tests:
308+
source:
309+
archive:
310+
file: https://github.com/NethermindEth/gas-benchmarks/actions/runs/23847558369/artifacts/6222084759
311+
pre_run_steps:
312+
- "perf-devnet-3/gas-bump.txt"
313+
- "perf-devnet-3/funding.txt"
314+
steps:
315+
setup:
316+
- "perf-devnet-3/setup/*.txt"
317+
test:
318+
- "perf-devnet-3/testing/*.txt"
319+
# Optional: External opcode metadata for the test suite.
320+
# When provided, opcode counts are included in the suite summary.json
321+
# and displayed in the UI opcode heatmap.
322+
opcodes: "opcodes_tracing.json"
323+
# Optional: Separate archive containing the opcodes file.
324+
# If not set, the opcodes file is searched in the main archive.
325+
# opcodes_file: https://github.com/NethermindEth/gas-benchmarks/actions/runs/23847558369/artifacts/6222074312
326+
```
327+
328+
| Option | Type | Required | Description |
329+
|--------|------|----------|-------------|
330+
| `file` | string | Yes | Local path or URL to a ZIP or tar.gz archive. GitHub Actions artifact URLs are auto-converted to API endpoints |
331+
| `pre_run_steps` | []string | No | Glob patterns for steps executed once before all tests |
332+
| `steps.setup` | []string | No | Glob patterns for setup phase files |
333+
| `steps.test` | []string | No | Glob patterns for test phase files |
334+
| `steps.cleanup` | []string | No | Glob patterns for cleanup phase files |
335+
| `opcodes` | string | No | Filename within the archive containing opcode count metadata (e.g., `opcodes_tracing.json`). The file must be a JSON object mapping test names to opcode counts: `{"test_name": {"OPCODE": count, ...}}` |
336+
| `opcodes_file` | string | No | Separate archive URL/path containing the opcodes file. If not set, the opcodes file is searched in the main `file` archive |
337+
338+
**GitHub Actions artifacts:** Browser URLs like `https://github.com/{owner}/{repo}/actions/runs/{run_id}/artifacts/{artifact_id}` are automatically converted to the GitHub API download endpoint. A GitHub token is required for artifact downloads (set via `runner.github_token` or `BENCHMARKOOR_RUNNER_GITHUB_TOKEN`).
339+
340+
**Archive extraction:** ZIP archives are extracted and any inner tarballs (common in GitHub Actions artifacts) are automatically extracted as well.
341+
302342
##### EEST Fixtures Source
303343

304344
EEST (Ethereum Execution Spec Tests) fixtures can be loaded from GitHub releases or GitHub Actions artifacts. This source type downloads fixtures from `ethereum/execution-spec-tests` and converts them to Engine API calls automatically.

pkg/config/config.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,8 @@ type LocalSourceV2 struct {
335335
// The file can be a local path or a URL (HTTP/HTTPS) to a ZIP or tar.gz archive.
336336
type ArchiveSourceConfig struct {
337337
File string `yaml:"file" mapstructure:"file"`
338+
OpcodesFile string `yaml:"opcodes_file,omitempty" mapstructure:"opcodes_file"`
339+
Opcodes string `yaml:"opcodes,omitempty" mapstructure:"opcodes"`
338340
PreRunSteps []string `yaml:"pre_run_steps,omitempty" mapstructure:"pre_run_steps"`
339341
Steps *StepsConfig `yaml:"steps,omitempty" mapstructure:"steps"`
340342
}

0 commit comments

Comments
 (0)