[smoke-test] Decrease retries from 16 to 4#19301
Open
gregnazario wants to merge 4 commits into
Open
Conversation
sionescu
approved these changes
Apr 1, 2026
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
JoshLind
reviewed
Apr 1, 2026
| # Reduced from 3 to 1: combined with SWARM_BUILD_NUM_RETRIES=1, gives 2×2=4 total | ||
| # attempts per test (was 4×4=16). Fewer retries reduce resource contention when | ||
| # multiple tests compete for ports on the same CI instance. | ||
| retries = 1 |
Contributor
There was a problem hiding this comment.
Silly question: if we reduce this, we won't allow a single smoke test flake anymore? Am I reading it correctly? 🤔
Contributor
Author
There was a problem hiding this comment.
It's 1 retry, so that's 2 tries
Contributor
There was a problem hiding this comment.
I see... 🤔 This may be too aggressive without fixing the flakes first 😄 (I worry we'll block folks and make them unhappy).
Will unblock this for you now, and take a look at the latest set of partitioned smoke test runs and try to fix any that I understand 🙏
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Remove tests that were explicitly marked as subsets of their epoch_changes counterparts. These tests were permanently #[ignore]'d and provided no additional coverage over the active variants. Removed from state_sync.rs: - test_fullnode_output_sync_no_epoch_changes - test_fullnode_execution_sync_no_epoch_changes - test_validator_sync_and_participate_no_epoch_changes - test_validator_fast_sync_and_participate_no_epoch_changes - test_validator_fast_sync_exponential_backoff_no_epoch_changes Removed from consensus_observer.rs: - test_consensus_observer_fast_sync_no_epoch_changes
The Move Prover z3 version check was silently failing when an existing z3 binary at $INSTALL_DIR couldn't report its version (e.g., wrong glibc, corrupt download, incompatible binary from CI runner image). This caused ~30% of unit test failures with: "cannot extract version from /home/runner/bin/z3" Changes: - Explicitly check if existing z3 binary can run and report version - Delete and reinstall broken binaries instead of skipping install - Add post-install verification to warn early on download corruption
gregnazario
force-pushed
the
greg/cleanup-testing
branch
from
9b2ae23 to
ab2312c
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
JoshLind
approved these changes
Apr 3, 2026
| # Reduced from 3 to 1: combined with SWARM_BUILD_NUM_RETRIES=1, gives 2×2=4 total | ||
| # attempts per test (was 4×4=16). Fewer retries reduce resource contention when | ||
| # multiple tests compete for ports on the same CI instance. | ||
| retries = 1 |
Contributor
There was a problem hiding this comment.
I see... 🤔 This may be too aggressive without fixing the flakes first 😄 (I worry we'll block folks and make them unhappy).
Will unblock this for you now, and take a look at the latest set of partitioned smoke test runs and try to fix any that I understand 🙏
Contributor
|
This issue is stale because it has been open 45 days with no activity. Remove the |
sionescu
approved these changes
May 20, 2026
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Contributor
✅ Forge suite
|
Contributor
✅ Forge suite
|
Contributor
✅ Forge suite
|
auto-merge was automatically disabled
May 21, 2026 00:29
Rebase failed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Reduces retries on testing, to bring out flaky tests / not spend so much time on them.
How Has This Been Tested?
Key Areas to Review
Type of Change
Which Components or Systems Does This Change Impact?
Checklist