Widen flaky poll timeouts in maintenance, STT, and instance-op integration tests#210
Open
sjainit wants to merge 1 commit into
Open
Widen flaky poll timeouts in maintenance, STT, and instance-op integration tests#210sjainit wants to merge 1 commit into
sjainit wants to merge 1 commit into
Conversation
…ation tests Several integration tests poll for cluster convergence and then assert on the outcome, but the polls use tight/default timeouts or wait on a different condition than the one asserted. Under CI scheduling contention a poll can return before the cluster converges, producing intermittent `expected:<true> but was:<false>` failures unrelated to the code under test. - TestClusterMaintenanceMode: re-point 13 polls at the exact asserted condition (maintenance-signal state, specific live-instance names, or the history entry's OPERATION_TYPE), assert the poll result instead of ignoring it, and wait TestHelper.WAIT_DURATION. - TestStateTransitionTimeout: assert the verifyByPolling result and use the 60s overload instead of the default 30s. - TestInstanceOperation: widen five 30s verifier polls to TestHelper.WAIT_DURATION (fixes the testEvacuateWithDisabledPartition flake seen in CI). Test-only change; no production code is modified. Widening is safe because a poll returns as soon as its predicate is true, so passing runs are unaffected and only slow-but-correct runs get the extra time they need. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sjainit
requested review from
LZD-PratyushBhatt,
PranaviAncha,
arkmish,
kabragaurav,
laxman-ch,
ngngwr and
thestreak101
as code owners
July 7, 2026 04:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issues
Fixes several pre-existing flaky integration tests that intermittently fail in CI with
expected:<true> but was:<false>, unrelated to the change under test. Most recentlytestEvacuateWithDisabledPartitionfailed this way on a PR CI run (Tests run: 1846, Failures: 1). No tracking issue is currently filed — happy to open one if preferred.Description
What. Test-only change that stabilizes three flaky integration-test classes by making their convergence polls wait long enough and assert the right condition.
Why. Each test polls for cluster convergence and then asserts on the outcome, but the polls use tight/default timeouts (30s) or — in the maintenance test — wait on a different condition than the one asserted. Under CI scheduling contention a poll can return before the cluster converges, yielding an intermittent
AssertionError: expected:<true> but was:<false>that has nothing to do with the change under test. For exampletestEvacuateWithDisabledPartitionpolled_admin.isEvacuateFinished(...)for only 30s; under load evacuation took longer (elapsed 36.8s) and the assert flaked.How.
TestClusterMaintenanceMode— re-point 13 polls at the exact asserted condition (maintenance-signal presence/absence, specific live-instance names, or the history entry'sOPERATION_TYPE), assert the poll result instead of ignoring it, and waitTestHelper.WAIT_DURATION. Some polls previously waited on stale/wrong live-instance counts; these are corrected to count-independent or name-based predicates.TestStateTransitionTimeout— assert theverifyByPollingresult and use the 60s overload instead of the default 30s.TestInstanceOperation— widen five 30sverifier(...)polls toTestHelper.WAIT_DURATION.Widening these budgets is safe: each poll returns as soon as its predicate becomes true, so a passing run is unaffected; only slow-but-correct runs get the extra time they need.
Tests
TestClusterMaintenanceMode,TestStateTransitionTimeout,TestInstanceOperation.mvn testresult:TestClusterMaintenanceModeandTestStateTransitionTimeoutwere validated locally under a CPU-throttle reproduction harness — both pass 3/3 iterations at the load that previously reproduced the flake, and pass at normal speed with no regression.TestInstanceOperationcompiles cleanly but is not locally reproducible in isolation (deepdependsOnMethodschain); relying on CI for that class.Changes that Break Backward Compatibility (Optional)
None. Test-only change; no production code, public API, config, or serialization-format changes.
Documentation (Optional)
None.
Commits
Code Quality
helix-styleformatting conventions.