Skip to content

Commit 18f31d4

Browse files
spencer-tbspencer-tb
committed
docs(releases): document fixture release process and tracks
1 parent 5e7e995 commit 18f31d4

1 file changed

Lines changed: 71 additions & 89 deletions

File tree

‎docs/running_tests/releases.md‎

Lines changed: 71 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,13 @@
11
# EEST Fixture Releases
22

3+
!!! warning "Two repositories — do not confuse them"
4+
- **`ethereum/execution-specs`** publishes Python *spec package* releases tagged `vX.Y.Z` (e.g. [`v2.20.0`](https://github.com/ethereum/execution-specs/releases/tag/v2.20.0)). **These contain no test fixtures** — they are the executable specification package only.
5+
- **Test fixtures** are published as **feature-scoped releases** on the same repository — `consensus@vX.Y.Z`, `<feat>-devnet@vX.Y.Z`, `benchmark@vX.Y.Z` — and are *not* attached to the `vX.Y.Z` package tags.
6+
- The legacy `fixtures_stable.tar.gz` / `fixtures_develop.tar.gz` artifacts (previously on `ethereum/execution-spec-tests`) are being retired in favour of the feature-scoped releases described here.
7+
38
## Formats and Release Layout
49

5-
@ethereum/execution-specs releases contain JSON test fixtures in various formats. Note that transaction type tests are executed directly from Python source using the [`execute`](./execute/index.md) command.
10+
Fixture releases contain JSON test fixtures in various formats. Note that transaction type tests are executed directly from Python source using the [`execute`](./execute/index.md) command.
611

712
| Format | Consumed by the client | Location in `.tar.gz` release |
813
| -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
@@ -12,116 +17,93 @@
1217
| [Transaction Tests](./test_formats/transaction_test.md) | - using a new simulator coming soon | None; executed directly from Python source,</br>using a release tag |
1318
| Blob Transaction Tests | - using the [eels/execute-blobs Simulator](./execute/hive.md#the-eelsexecute-blobs-simulator) and | None; executed directly from Python source,</br>using a release tag |
1419

15-
## Fixture Output Directory Structure
16-
17-
Inside each format directory, fixtures are grouped by **target fork**.
18-
19-
The top-level subdirectory identifies the fork **under test**. Below it,
20-
fixtures mirror the `./tests/` source layout: each directory corresponds
21-
to the fork where the functionality was originally introduced. Because
22-
tests declare `valid_from`, a single target fork directory contains
23-
fixtures from every prior fork whose tests are still valid at that fork.
24-
25-
### Consensus fixture layout
26-
27-
```text
28-
fixtures/
29-
└── blockchain_tests/
30-
├── for_prague/ # filled targeting Prague
31-
│ ├── istanbul/ # tests introduced in Istanbul
32-
│ │ └── eip1344_chainid/...
33-
│ ├── cancun/ # tests introduced in Cancun
34-
│ │ └── eip4844_blobs/...
35-
│ └── prague/ # tests introduced in Prague
36-
│ └── eip7702_set_code_tx/...
37-
└── for_osaka/ # filled targeting Osaka
38-
├── istanbul/
39-
│ └── eip1344_chainid/...
40-
├── cancun/
41-
│ └── eip4844_blobs/...
42-
├── prague/
43-
│ └── eip7702_set_code_tx/...
44-
└── osaka/ # tests introduced in Osaka
45-
└── eip7692_eof_v1/...
46-
```
20+
## Release Tracks
4721

48-
Other format directories (`state_tests/`, `blockchain_tests_engine/`)
49-
follow the same layout.
22+
Fixtures are released on independent tracks. Each track has its own tag namespace, artifact, and cadence.
5023

51-
### Benchmark fixture layout
24+
| Track | Tag | Artifact | Scope | Built from |
25+
| --------- | -------------------- | ------------------------------- | ------------------------------------------------------------------ | ----------------- |
26+
| Consensus | `consensus@vX.Y.Z` | `fixtures_consensus.tar.gz` | All forks, all tests (including legacy tests) | `main`/`master` |
27+
| Devnet | `<feat>-devnet@vX.Y.Z` | `fixtures_<feat>-devnet.tar.gz` | All forks, all tests, for an upcoming-fork feature under active devnet testing | the devnet branch |
28+
| Benchmark | `benchmark@vX.Y.Z` | `fixtures_benchmark.tar.gz` | EVM benchmarking tests | `main`/`master` |
5229

53-
When filling with `--gas-benchmark-values`, benchmark tests additionally
54-
include the gas limit in the subdirectory name (`for_{fork}_at_{gas}M`,
55-
where `{gas}` is in millions, zero-padded to four digits), with one
56-
subdirectory per gas value:
30+
- **Consensus** releases track clients' production branches and are tagged frequently (roughly once or twice a week). They are the "must pass" release for mainnet CI.
31+
- **Devnet** releases target a specific feature under active development (e.g. `bal-devnet`). They are still WIP and may not contain full coverage for all EIPs; see the corresponding release notes for the coverage provided.
32+
- **Benchmark** (and, in future, zkEVM) releases are produced separately for their specialized consumers.
5733

58-
```text
59-
fixtures/
60-
└── blockchain_tests/
61-
├── for_osaka_at_0001M/ # 1M gas benchmark
62-
│ └── benchmark/compute/...
63-
└── for_osaka_at_0002M/ # 2M gas benchmark
64-
└── benchmark/compute/...
65-
```
34+
## Versioning Scheme
6635

67-
When filling with `--fixed-opcode-count`, the opcode count replaces the
68-
gas limit in the subdirectory name (`for_{fork}_at_opcount_{N}K`, where
69-
`{N}` is in thousands and may include decimals):
70-
71-
```text
72-
fixtures/
73-
└── blockchain_tests/
74-
├── for_osaka_at_opcount_10K/ # 10K opcodes
75-
│ └── benchmark/compute/...
76-
└── for_osaka_at_opcount_20K/ # 20K opcodes
77-
└── benchmark/compute/...
78-
```
36+
Tags use the form `<track>@v<X>.<Y>.<Z>`. The underlying git tag is prefixed with `tests-` (e.g. `tests-consensus@v1.2.3`); the GitHub release title omits the prefix (`consensus@v1.2.3`).
37+
38+
The meaning of `X.Y.Z` depends on the track:
7939

80-
## Release URLs and Tarballs
40+
| Component | Consensus track | Devnet track |
41+
| --------- | --------------------------- | --------------------------- |
42+
| `X` | Fork number | Devnet version |
43+
| `Y` | Spec/test change → a change in behaviour | Spec/test change → a change in behaviour |
44+
| `Z` | New tests only (no behaviour change) | New tests only (no behaviour change) |
8145

82-
### Versioning Scheme
46+
This keeps the version purely ordered within a track: a higher `X.Y.Z` on the same track is always the newer release.
8347

84-
EEST framework and test sources and fixture releases are tagged use a semantic versioning scheme, `<optional:<pre_release_name@>>v<MAJOR>.<MINOR>.<PATCH>` as following:
48+
## Pinning Guidance
8549

86-
- `<MAJOR>`: An existing fixture format has changed (potentially breaking change). Action must be taken by client teams to ensure smooth upgrade to the new format.
87-
- `<MINOR>`: Additional coverage (new tests, or a new format) have been added to the release.
88-
- `<PATCH>`: A bug-fix release; an error in the tests or framework has been patched.
50+
Mapped to a typical client CI setup:
8951

90-
Please see below for an explanation of the optional `<pre_release_name>` that is used in pre-releases.
52+
- **Blocking gate (current + past forks).** Pin a specific `consensus@vX.Y.Z` for reproducible, no-rug-pull CI on your `master`/production branch, or follow the latest consensus release if a moving target is acceptable. This supersedes the old `fixtures_develop`/`fixtures_stable` artifacts.
53+
- **Non-blocking gate (next fork).** Use the current `<feat>-devnet@vX.Y.Z` release for the upcoming fork's active devnet (e.g. `bal-devnet@vX.Y.Z`). Treat it as advisory — devnet coverage changes rapidly and should not block merges.
9154

92-
### Standard Releases
55+
!!! note "Devnet vs. consensus overlap"
56+
Devnet releases are filled for all forks/tests, so they overlap with the consensus release. If your blocking gate already runs a consensus release, the devnet gate re-runs that shared coverage. Deduplicating that overlap is consumer-side concern handled when resolving/consuming releases.
9357

94-
Releases are published on the @ethereum/execution-specs [releases](https://github.com/ethereum/execution-specs/releases) page. Standard releases are tagged using the format `vX.Y.Z` (they don't have a `<pre_release_name>`).
58+
## Creating a Fixture Release
9559

96-
For standard releases, two tarballs are available:
60+
Fixture releases are produced by manually dispatching the [`release_fixtures.yaml`](https://github.com/ethereum/execution-specs/blob/master/.github/workflows/release_fixtures.yaml) workflow. There is no tag to push by hand: the workflow builds the fixtures and, only on success, creates the tag and the (draft) GitHub release.
9761

98-
| Release Artifact | Fork/feature scope |
99-
| ------------------------- | ----------------------------------------------------------------------- |
100-
| `fixtures_stable.tar.gz` | Tests for all forks up to and including the last deployed ("stable") mainnet fork ("must pass") |
101-
| `fixtures_develop.tar.gz` | Tests for all forks up to and including the last development fork |
62+
```bash
63+
gh workflow run release_fixtures.yaml -f feature=<feature>@vX.Y.Z [-f branch=<branch>]
64+
```
10265

103-
I.e., `fixtures_develop` are a superset of `fixtures_stable`.
66+
### Inputs
10467

105-
!!! tip "Release features opt into all fixture formats via `feature.yaml`"
106-
Tarball output (`.tar.gz`) does not by itself include the pre-allocation group formats (`BlockchainEngineXFixture`, `BlockchainEngineStatefulFixture`). A release feature requests them by adding `--generate-all-formats` to its `fill-params` in `.github/configs/feature.yaml`:
107-
```console
108-
uv run fill --generate-all-formats --output=fixtures_stable.tar.gz tests/
109-
```
68+
| Input | Required | Description |
69+
| --------- | ----------------- | ---------------------------------------------------------------------------------------------------- |
70+
| `feature` | yes | `<feature>@vX.Y.Z` (the `tests-` prefix is optional and is added automatically to the tag). |
71+
| `branch` | for `*-devnet` | Branch to build and release from. Optional for non-devnet features; **required** for devnet releases. |
11072

111-
### Pre-Release and Devnet Releases
73+
`<feature>` must be a key in [`.github/configs/feature.yaml`](https://github.com/ethereum/execution-specs/blob/master/.github/configs/feature.yaml) (e.g. `consensus`, `benchmark`), or a `<feat>-devnet` name that resolves to the shared `devnet` feature.
11274

113-
Intermediate releases that target specific subsets of features or tests under active development are published at @ethereum/execution-specs [releases](https://github.com/ethereum/execution-specs/releases).
75+
### Devnet releases
11476

115-
These releases are tagged using the format `<pre_release_name>@vX.Y.Z`.
77+
Devnet releases must be named `<feat>-devnet@vX.Y.Z` (e.g. `bal-devnet@v1.0.0`) and must specify the branch to release from:
11678

117-
<!-- TODO: These example tags live in the legacy @ethereum/execution-spec-tests repo; replace with current @ethereum/execution-specs examples once available. -->
118-
Examples:
79+
```bash
80+
gh workflow run release_fixtures.yaml -f feature=bal-devnet@v1.0.0 -f branch=bal-devnet-7
81+
```
82+
83+
A bare `devnet@vX.Y.Z` (no `<feat>-` prefix), or a `*-devnet` release without a `branch`, fails fast in the first job before any fixtures are built.
84+
85+
### What the workflow produces
86+
87+
On success the workflow:
88+
89+
1. Builds `fixtures_<feature>.tar.gz` for the resolved feature (per its `evm-type` and `fill-params` in `feature.yaml`).
90+
2. Creates the git tag `tests-<feature>@vX.Y.Z` on the released commit (the `branch` HEAD when given, otherwise the dispatch commit).
91+
3. Publishes a **draft pre-release** to [`ethereum/execution-specs`](https://github.com/ethereum/execution-specs/releases), titled `<feature>@vX.Y.Z` (no `tests-` prefix), with the fixture tarball(s) attached.
11992

120-
- [`fusaka-devnet-1@v1.0.0`](https://github.com/ethereum/execution-spec-tests/releases/tag/fusaka-devnet-1%40v1.0.0) - this fixture release contains tests adhering to the [Fusaka Devnet 1 spec](https://notes.ethereum.org/@ethpandaops/fusaka-devnet-1).
121-
- [`benchmark@v0.0.3`](https://github.com/ethereum/execution-spec-tests/releases/tag/benchmark%40v0.0.3) - this fixture release contains tests specifically aimed at benchmarking EVMs.
93+
| Example dispatch | Git tag | Release title | Artifact |
94+
| ---------------- | ------- | ------------- | -------- |
95+
| `feature=consensus@v1.2.3` | `tests-consensus@v1.2.3` | `consensus@v1.2.3` | `fixtures_consensus.tar.gz` |
96+
| `feature=bal-devnet@v1.0.0 branch=bal-devnet-7` | `tests-bal-devnet@v1.0.0` | `bal-devnet@v1.0.0` | `fixtures_bal-devnet.tar.gz` |
12297

123-
Devnet releases should be treated as WIP and may not yet contain full test coverage (or even coverage for all EIPs). The coverage provided by these releases is detailed in the corresponding release notes.
98+
The release is created as a draft; review and publish it from the GitHub releases page.
99+
100+
!!! tip "Release features opt into all fixture formats via `feature.yaml`"
101+
Tarball output (`.tar.gz`) does not by itself include the pre-allocation group formats (`BlockchainEngineXFixture`, `BlockchainEngineStatefulFixture`). A release feature requests them by adding `--generate-all-formats` to its `fill-params` in `.github/configs/feature.yaml`:
102+
```console
103+
# Automatically enables --generate-all-formats due to .tar.gz output
104+
uv run fill --output=fixtures_consensus.tar.gz tests/
105+
```
124106

125-
### Help Downloading Releases
107+
## Help Downloading Releases
126108

127109
The [`consume cache`](./consume/cache.md) command can be used to resolve EEST release and pre-release tags to release URLs and download them.

0 commit comments

Comments
 (0)