refactor(end-to-end): promote scenario to commands, switch epoch length based wait duration, rework and cleanup assertions by Alenar · Pull Request #3316 · IntersectMBO/mithril

Alenar · 2026-06-08T15:35:21Z

Content

This PR includes three major changes to the end-to-end runners:

Scenario are now clap sub-commands, meaning to run run-only you need to pass run-only instead of --run-only and it have to be passed at the end of the command
All wait have been changed from hardcoded values to multiple for epoch length based duration like "a tenth of an epoch" (computed from the cardano_slot_length and cardano_epoch_length args)
The assertions module, which contains shared, have been renamed to toolkit and overhauled from an list of static method into a tree of structures (split by subdomains) to allow them to hold a context.

Additional details

About the command line arguments:

Args struct have been renamed to Cli to avoid a collision with clap::Args
most of the Cli values they have been split into multiple structures:
- NetworkTopologyArgs: number of aggregators, signers, do we use the mithril relay, the dmq, ...
- MithrilArgs: Mithril state machines run interval, the start era, ...
- CardanoDevnetArgs: Devnet scripts location, Cardano node version, slot/epoch lenght, ...
- they are flattened into the Cli struct, meaning there's no functional change for them for users

About scenario:

their "scripts" have been moved to a new scenario folder module
if no scenario command subcommand is given, Full is used
some arguments have been moved from the main argument list to the subcommands:
- to Full: check_client_cli_snapshot_converter has it was only used by that scenario, signed_entity_types has the upcoming minimal scenario won't allow for more than one
- to RunOnly: signed_entity_types same cause as in the Full scenario

About the refactor of the assertion module into toolkit:

scoping check and wait into struct allowed named simplification (e.g. no need to repeat "check" in method names has you will use them through their structures: toolkit.check.certificate.is_creating_certificate_with_enough_signers instead of toolkit.check.certificate.check_is_creating_certificate_with_enough_signers)
new wait_for_latest_artifact utility method was added to remove duplication between wait_for_artifact methods in each signed entity type dedicated check toolkit
To avoid duplication with the upcoming minimal scenario, check steps of each signed entities have been aggregated into is_certified_and_verified methods
Methods that were fetching artifact individually based on a hash have been removed, as their value was low, the only additional check that they were doing (asserting a minimal value for artifact epoch) have been transferred to new check_artifact methods
check_artifact naming is willingly broad to allow future addition of additional checks

Pre-submit checklist

Branch
- Tests are provided (if possible)
- Crates versions are updated (if relevant)
- Commit sequence broadly makes sense
- Key commits have useful messages
PR
- All check jobs of the CI have succeeded
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested
Documentation
- Update README file (if relevant)
- No new TODOs introduced

Issue(s)

Relates to #3151

github-actions · 2026-06-08T15:50:59Z

Test Results

5 files ±0 206 suites ±0 1h 3m 25s ⏱️ - 1m 23s
3 156 tests +3 3 156 ✅ +3 0 💤 ±0 0 ❌ ±0
10 628 runs +9 10 628 ✅ +9 0 💤 ±0 0 ❌ ±0

Results for commit 0e64a24. ± Comparison against base commit 8c06fae.

♻️ This comment has been updated with latest results.

Copilot

Pull request overview

This PR refactors the Mithril end-to-end test runner to make scenarios explicit clap subcommands, to base polling/wait timings on the configured Cardano epoch duration, and to replace the previous flat assertions helpers with a structured toolkit that can carry shared context across checks/exec/waits.

Changes:

Promote scenarios to clap subcommands (full, run-only) and rework CLI argument grouping/flattening accordingly.
Replace hardcoded wait durations with epoch-derived delays via a shared ScenarioToolkitContext / AttemptPolicy.
Rename/replace assertions with a structured toolkit (check/exec/wait), and update scenarios/infrastructure to use it.

Reviewed changes

Copilot reviewed 27 out of 28 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
mithril-test-lab/mithril-end-to-end/src/toolkit/wait.rs	New wait helpers using epoch-derived delays and `attempt!`.
mithril-test-lab/mithril-end-to-end/src/toolkit/mod.rs	Introduces `ScenarioToolkit` facade that groups check/exec/wait toolkits.
mithril-test-lab/mithril-end-to-end/src/toolkit/exec.rs	New exec helpers (bootstrap genesis, protocol params update, era marker, etc.).
mithril-test-lab/mithril-end-to-end/src/toolkit/context.rs	Adds `ScenarioToolkitContext` + `AttemptPolicy` to compute epoch-based delays (with unit tests).
mithril-test-lab/mithril-end-to-end/src/toolkit/check/utils.rs	Shared HTTP/attempt helpers for “wait for latest artifact” + common assertions.
mithril-test-lab/mithril-end-to-end/src/toolkit/check/toolkit.rs	New `CheckToolkit` grouping artifact verification routines and client checks.
mithril-test-lab/mithril-end-to-end/src/toolkit/check/mod.rs	New module layout for toolkit checks.
mithril-test-lab/mithril-end-to-end/src/toolkit/check/mithril_stake_distribution.rs	Toolkit-based checks for Mithril stake distribution artifact verification.
mithril-test-lab/mithril-end-to-end/src/toolkit/check/certificate.rs	Toolkit-based checks for certificate creation with expected signer count.
mithril-test-lab/mithril-end-to-end/src/toolkit/check/cardano_transactions.rs	Toolkit-based checks for Cardano transactions artifact verification.
mithril-test-lab/mithril-end-to-end/src/toolkit/check/cardano_stake_distribution.rs	Toolkit-based checks for Cardano stake distribution artifact verification (epoch/hash downloads).
mithril-test-lab/mithril-end-to-end/src/toolkit/check/cardano_database.rs	Toolkit-based checks for Cardano DB snapshot + digests map + client verification.
mithril-test-lab/mithril-end-to-end/src/toolkit/check/cardano_blocks_transactions.rs	Toolkit-based checks for block/tx certification via client CLI (v2).
mithril-test-lab/mithril-end-to-end/src/scenario/run_only.rs	Converts “run-only” flow to `RunOnlyScenario` using `ScenarioToolkit`.
mithril-test-lab/mithril-end-to-end/src/scenario/mod.rs	Exposes scenario types (`FullScenario`, `RunOnlyScenario`).
mithril-test-lab/mithril-end-to-end/src/scenario/full.rs	Refactors former “Spec” flow into `FullScenario` and routes checks through toolkits.
mithril-test-lab/mithril-end-to-end/src/mithril/infrastructure.rs	Threads `ScenarioToolkit` into infrastructure for era-marker registration and removes old flags.
mithril-test-lab/mithril-end-to-end/src/main.rs	Major CLI refactor: scenario subcommands, flattened arg groups, and scenario dispatch logic.
mithril-test-lab/mithril-end-to-end/src/lib.rs	Exposes `scenario` and `toolkit` modules; removes old `assertions` exports.
mithril-test-lab/mithril-end-to-end/src/assertions/wait.rs	Removes old wait assertions (migrated into toolkit).
mithril-test-lab/mithril-end-to-end/src/assertions/mod.rs	Removes old assertions module entrypoint.
mithril-test-lab/mithril-end-to-end/src/assertions/exec.rs	Removes old exec assertions (migrated into toolkit).
mithril-test-lab/mithril-end-to-end/src/assertions/check.rs	Removes old check assertions (migrated into toolkit).
mithril-test-lab/mithril-end-to-end/Cargo.toml	Bumps crate version to 0.5.0.
mithril-test-lab/cardano-devnet/README.md	Updates documentation to use `run-only` subcommand instead of `--run-only`.
Cargo.lock	Updates lockfile for the `mithril-end-to-end` version bump.
.github/workflows/ci.yml	Updates CI E2E invocation to include the `full` scenario subcommand.
.github/workflows/backward-compatibility.yml	Updates backward-compat workflow E2E invocation to include the `full` scenario subcommand.

turmelclem

LGTM 👍

* rename `Args` to `Cli` to avoid name colision with `clap::Args` * restructured args are flatten in to the main `Cli` struct, meaning that there's no changes to the cli interface

…e for clarity * introduced a new `scenario` module to group related logic * moved `Spec` to `FullScenario` and `RunOnly` to `RunOnlyScenario`

First step of the refactor which will move those functions into a type which will hold the test context.

…heckToolkit`, `ExecToolkit`, `WaitToolkit`) * Added `ScenarioToolkit` for centralized test context handling. * Implemented `ScenarioToolkitContext` and `AttemptPolicy` to control retry policies and delays.

* Moved toolkit functions into their dedicated struct. * Improved reusability and modularity by integrating context handling. * Temporary keep old methods, with an indirection, to keep compatibity and avoid a bigbang

…ctor

…cat types + certificate Integrated new specialized toolkits (`CheckCardanoBlocksTransactionsToolkit`, `CheckCardanoDatabaseToolkit`, `CheckCardanoStakeDistributionToolkit`, `CheckCardanoTransactionsToolkit`, `CheckCertificateToolkit`, `CheckMithrilStakeDistributionToolkit`) to enhance modularity and reusability in end-to-end testing scenarios.

…toolkits * Add "all in one" `is_certified_and_verified` routines across toolkits that checks artificats productions, certificates signing, and client verification * Streamlined `client_*` verification methods into consistent naming (`verify_with_client`/`verify_transactions_with_client`/`verify_blocks_with_client`).

…ructure - Replaced standalone toolkit function calls with corresponding methods in `ScenarioToolkit`. - Updated `FullScenario` and `RunOnlyScenario` to use `ScenarioToolkit` for modular subcomponents. - Removed obsolete standalone toolkit methods.

Unified artifact-related methods across check toolkits by replacing repetitive methods with reusable `wait_for_artifact` and `check_artifact` utilities.

…edicated module

…t` across check toolkits

`is_creating_certificate_with_enough_signers` wait for any certificate, not for a certificate with a given minimum of signers.

…uration utilities - Remove `Default` trait implementations across various toolkits to force user to provide an epoch length - Refactored `ScenarioToolkitContext` to support Cardano epoch-based duration calculations with new helper methods and constructors.

…tilities - Updated all `attempt!` calls to use `ScenarioToolkitContext` delay helpers (`tenth_epoch_delay` and `half_epoch_delay`) for consistency and configurability.

- `Full` and `RunOnly` Scenarios are now subcommands - Default scenario to `full` - Moved `check_client_cli_snapshot_converter` to full subcommand as it's only relevant here - Moved `signed_entity_types` to both full and runonly subcommands (note: upcoming "minimal" scenario will only allow one signed entity type, this give the flexibility to parametrize that) - Simplified infrastructure and CLI parsing logic by removing redundant arguments. - Updated `README.md` and CI to reflect changes in Mithril execution commands.

… consistency Has those functions are in the `WaitContext` we do not need to repeat `wait_` in their names.

jpraynaud

LGTM 🚀

Alenar self-assigned this Jun 8, 2026

Alenar requested review from jpraynaud and turmelclem as code owners June 8, 2026 15:35

Alenar added CI/CD 🏭 CI/CD testing 🔁 Something related to tests labels Jun 8, 2026

Alenar temporarily deployed to testing-preview June 8, 2026 15:53 — with GitHub Actions Inactive

jpraynaud requested a review from Copilot June 8, 2026 15:54

Copilot started reviewing on behalf of jpraynaud June 8, 2026 15:54 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Alenar force-pushed the djo/3151/e2e-refactor-to-prepare-snark-scenario branch 2 times, most recently from 3201452 to 22cb8ae Compare June 8, 2026 16:24

Alenar temporarily deployed to testing-preview June 8, 2026 16:43 — with GitHub Actions Inactive

turmelclem approved these changes Jun 9, 2026

View reviewed changes

Alenar added 16 commits June 9, 2026 14:29

refactor(e2): restructure Cli arguments into modular substructures

792f3c1

* rename `Args` to `Cli` to avoid name colision with `clap::Args` * restructured args are flatten in to the main `Cli` struct, meaning that there's no changes to the cli interface

refactor(e2e): restructure scenarios into scenario module and renam…

815ea86

…e for clarity * introduced a new `scenario` module to group related logic * moved `Spec` to `FullScenario` and `RunOnly` to `RunOnlyScenario`

refactor(e2e): rename assertions module to toolkit

e45e30a

First step of the refactor which will move those functions into a type which will hold the test context.

feat(e2e): introduce ScenarioToolkit with modular subcomponents (`C…

246ecb7

…heckToolkit`, `ExecToolkit`, `WaitToolkit`) * Added `ScenarioToolkit` for centralized test context handling. * Implemented `ScenarioToolkitContext` and `AttemptPolicy` to control retry policies and delays.

refactor(e2e): consolidate assertions into ****Toolkit structs

2892647

* Moved toolkit functions into their dedicated struct. * Improved reusability and modularity by integrating context handling. * Temporary keep old methods, with an indirection, to keep compatibity and avoid a bigbang

refactor(e2e): promote check.rs to module directory for future refa…

f8f0235

…ctor

refactor(e2e): simplify artifact handling in check toolkits

9bce508

Unified artifact-related methods across check toolkits by replacing repetitive methods with reusable `wait_for_artifact` and `check_artifact` utilities.

refactor(e2e): move struct and utils from toolkit/check/mod.rs to d…

6f0ad4b

…edicated module

refactor(e2e): rename wait_for_artifact to `wait_for_latest_artifac…

ba16503

…t` across check toolkits

refactor(e2e): fix logging message in certificate check utility

0c95e9d

`is_creating_certificate_with_enough_signers` wait for any certificate, not for a certificate with a given minimum of signers.

refactor(e2e): replace hardcoded durations with context-based delay u…

5fd7459

…tilities - Updated all `attempt!` calls to use `ScenarioToolkitContext` delay helpers (`tenth_epoch_delay` and `half_epoch_delay`) for consistency and configurability.

refactor(e2e): rename wait_for_* methods to improve readability and…

65e0097

… consistency Has those functions are in the `WaitContext` we do not need to repeat `wait_` in their names.

jpraynaud approved these changes Jun 9, 2026

View reviewed changes

Comment thread .github/workflows/backward-compatibility.yml

Comment thread mithril-test-lab/mithril-end-to-end/src/main.rs

Alenar added 2 commits June 9, 2026 16:47

feat(e2e): logs command line arguments at startup

1193511

chore: bump mithril-end-to-end from 0.4.139 to 0.5.0

0e64a24

Alenar force-pushed the djo/3151/e2e-refactor-to-prepare-snark-scenario branch from 22cb8ae to 0e64a24 Compare June 9, 2026 14:47

Alenar temporarily deployed to testing-preview June 9, 2026 15:18 — with GitHub Actions Inactive

Alenar merged commit 905fe2c into main Jun 9, 2026
61 checks passed

Alenar deleted the djo/3151/e2e-refactor-to-prepare-snark-scenario branch June 9, 2026 15:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(end-to-end): promote scenario to commands, switch epoch length based wait duration, rework and cleanup assertions - #3316

refactor(end-to-end): promote scenario to commands, switch epoch length based wait duration, rework and cleanup assertions#3316
Alenar merged 19 commits into
mainfrom
djo/3151/e2e-refactor-to-prepare-snark-scenario

Alenar commented Jun 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

turmelclem left a comment

Uh oh!

jpraynaud left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Alenar commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content

Additional details

Pre-submit checklist

Issue(s)

Uh oh!

github-actions Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

turmelclem left a comment

Choose a reason for hiding this comment

Uh oh!

jpraynaud left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Alenar commented Jun 8, 2026 •

edited

Loading

github-actions Bot commented Jun 8, 2026 •

edited

Loading