Skip to content

fix(e2e): remove epoch gap flakiness - #3445

Open
jpraynaud wants to merge 5 commits into
mainfrom
jpraynaud/fix-e2e-flakiness
Open

fix(e2e): remove epoch gap flakiness#3445
jpraynaud wants to merge 5 commits into
mainfrom
jpraynaud/fix-e2e-flakiness

Conversation

@jpraynaud

@jpraynaud jpraynaud commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Content

This PR includes the removal of several sources of flakiness in the e2e tests which caused intermittent Timeout exhausted waiting for Certificate for epoch XX failures in the CI:

  • Poll epoch transitions at a tenth of the epoch duration, so that target epochs are detected shortly after their boundary instead of up to a full epoch late with the previous exponential backoff
  • Certify the current epoch before the protocol parameters restart of the aggregator, so that the restart can no longer create an unrecoverable epoch gap
  • Extend the artifact production timeout from one to three epochs: an artifact type is produced once per epoch, so a wait started late in an epoch may need to span a full production cycle before the awaited artifact exists, which a single epoch timeout could not cover
  • Lighten the updated protocol parameters by reducing the number of lotteries played, closer to the initial parameter set, as the test only needs to exercise the parameters update

The fixed e2e test has been run 381 times with 1 failure, a 99.74% success ratio.

Pre-submit checklist

  • Branch
    • Crates versions are updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested

Issue(s)

Closes #3452

@jpraynaud jpraynaud self-assigned this Jul 30, 2026
@github-actions

github-actions Bot commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Results

     5 files  ±0     209 suites  ±0   57m 19s ⏱️ - 2m 11s
 3 331 tests +1   3 331 ✅ +1  0 💤 ±0  0 ❌ ±0 
11 093 runs  +3  11 093 ✅ +3  0 💤 ±0  0 ❌ ±0 

Results for commit 520f12b. ± Comparison against base commit 71ea932.

♻️ This comment has been updated with latest results.

@jpraynaud
jpraynaud temporarily deployed to testing-preview July 30, 2026 18:00 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-2-preview July 30, 2026 18:00 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-preview July 30, 2026 19:23 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-2-preview July 30, 2026 19:23 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-preview July 30, 2026 20:45 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-2-preview July 30, 2026 20:45 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-2-preview July 31, 2026 07:58 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-preview July 31, 2026 07:58 — with GitHub Actions Inactive
@jpraynaud
jpraynaud force-pushed the jpraynaud/fix-e2e-flakiness branch 4 times, most recently from 4847826 to 4f65f57 Compare August 3, 2026 06:30
@jpraynaud
jpraynaud temporarily deployed to testing-preview August 3, 2026 06:48 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-2-preview August 3, 2026 06:48 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-preview August 3, 2026 10:43 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-2-preview August 3, 2026 10:43 — with GitHub Actions Inactive
@jpraynaud
jpraynaud force-pushed the jpraynaud/fix-e2e-flakiness branch from 4f65f57 to 8ad0e8c Compare August 3, 2026 14:59
@jpraynaud
fix(e2e): poll epoch transitions at a tenth of the epoch duration
92331fe 
The exponential poll backoff exceeded short devnet epochs, so target
epochs could be detected up to a full epoch after their boundary.
@jpraynaud
fix(e2e): certify current epoch before protocol parameters restart
92d5641 
A restart crossing an epoch boundary before the current epoch is
certified creates an unrecoverable epoch gap that blocks the aggregator.
@jpraynaud
fix(e2e): extend artifact production timeout to three epochs
9bb7e7d 
A single epoch left no slack for certificate sealing latency, causing
spurious 'Timeout exhausted waiting for...' failures.
@jpraynaud
fix(e2e): lighten updated protocol parameters
31508be 
Reduce the number of lotteries played to get closer to the initial
parameter set, as the test only needs to exercise the parameters update.
@jpraynaud
chore: upgrade crate versions
520f12b 
* mithril-end-to-end from `0.5.9` to `0.5.10`
@jpraynaud
jpraynaud force-pushed the jpraynaud/fix-e2e-flakiness branch from 8ad0e8c to 520f12b Compare August 3, 2026 15:51
@jpraynaud
jpraynaud marked this pull request as ready for review August 3, 2026 15:52
@jpraynaud
jpraynaud requested a review from Alenar as a code owner August 3, 2026 15:52
Copilot AI review requested due to automatic review settings August 3, 2026 15:52
@jpraynaud
jpraynaud requested a review from turmelclem as a code owner August 3, 2026 15:52
Copilot AI reviewed Aug 3, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces intermittent e2e CI failures caused by epoch-boundary timing and aggregator restarts by tightening epoch transition polling, ensuring certificate continuity across restarts, and increasing artifact wait timeouts.

Changes:

  • Switch epoch-transition waiting to a constant “tenth of epoch” polling cadence.
  • Ensure the current epoch is certified before restarting the aggregator for protocol-parameter updates.
  • Increase artifact production timeout to cover up to three epochs and reduce concatenation protocol parameters used during the update.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
mithril-test-lab/mithril-end-to-end/src/toolkit/wait.rs Uses the new constant tenth-of-epoch backoff when waiting for target epoch transitions.
mithril-test-lab/mithril-end-to-end/src/toolkit/exec.rs Lightens concatenation protocol parameters used during the update/restart step.
mithril-test-lab/mithril-end-to-end/src/toolkit/context.rs Adds tenth_of_epoch_poll_backoff() and increases artifact production timeout to 3 epochs (plus tests).
mithril-test-lab/mithril-end-to-end/src/scenario/full.rs Adds a pre-restart certification check to prevent unrecoverable epoch gaps.
mithril-test-lab/mithril-end-to-end/Cargo.toml Bumps mithril-end-to-end crate version to 0.5.10.
Cargo.lock Updates lockfile for the mithril-end-to-end version bump.

Comment thread mithril-test-lab/mithril-end-to-end/src/scenario/full.rs
// A certificate chain lagging one epoch behind the chain tip is normal, but a restart
// crossing an epoch boundary before the current epoch is certified would create an
// unrecoverable epoch gap and block the aggregator
let current_epoch = chain_observer.get_current_epoch().await?.unwrap_or_default();
Comment thread mithril-test-lab/mithril-end-to-end/src/toolkit/context.rs
Comment on lines +37 to +38
let delay = self.timeout_for_epochs(1) / 10;
Backoff::new(delay, delay, 1)
@jpraynaud
jpraynaud temporarily deployed to testing-2-preview August 3, 2026 16:09 — with GitHub Actions Inactive
@jpraynaud
jpraynaud temporarily deployed to testing-preview August 3, 2026 16:09 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@Alenar Alenar Awaiting requested review from Alenar Alenar is a code owner
@turmelclem turmelclem Awaiting requested review from turmelclem turmelclem is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@jpraynaud jpraynaud

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Fix flakiness of e2e tests in the CI

2 participants

@jpraynaud