Skip to content

Widen flaky poll timeouts in maintenance, STT, and instance-op integration tests#210

Open
sjainit wants to merge 1 commit into
linkedin:devfrom
sjainit:sjainit/fix-flaky-integration-tests
Open

Widen flaky poll timeouts in maintenance, STT, and instance-op integration tests#210
sjainit wants to merge 1 commit into
linkedin:devfrom
sjainit:sjainit/fix-flaky-integration-tests

Conversation

@sjainit

@sjainit sjainit commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Issues

  • My PR addresses the following Helix issues and references them in the PR description:

Fixes several pre-existing flaky integration tests that intermittently fail in CI with expected:<true> but was:<false>, unrelated to the change under test. Most recently testEvacuateWithDisabledPartition failed this way on a PR CI run (Tests run: 1846, Failures: 1). No tracking issue is currently filed — happy to open one if preferred.

Description

  • Here are some details about my PR:

What. Test-only change that stabilizes three flaky integration-test classes by making their convergence polls wait long enough and assert the right condition.

Why. Each test polls for cluster convergence and then asserts on the outcome, but the polls use tight/default timeouts (30s) or — in the maintenance test — wait on a different condition than the one asserted. Under CI scheduling contention a poll can return before the cluster converges, yielding an intermittent AssertionError: expected:<true> but was:<false> that has nothing to do with the change under test. For example testEvacuateWithDisabledPartition polled _admin.isEvacuateFinished(...) for only 30s; under load evacuation took longer (elapsed 36.8s) and the assert flaked.

How.

  • TestClusterMaintenanceMode — re-point 13 polls at the exact asserted condition (maintenance-signal presence/absence, specific live-instance names, or the history entry's OPERATION_TYPE), assert the poll result instead of ignoring it, and wait TestHelper.WAIT_DURATION. Some polls previously waited on stale/wrong live-instance counts; these are corrected to count-independent or name-based predicates.
  • TestStateTransitionTimeout — assert the verifyByPolling result and use the 60s overload instead of the default 30s.
  • TestInstanceOperation — widen five 30s verifier(...) polls to TestHelper.WAIT_DURATION.

Widening these budgets is safe: each poll returns as soon as its predicate becomes true, so a passing run is unaffected; only slow-but-correct runs get the extra time they need.

Tests

  • Test-only PR (no product code touched). Changed classes:

TestClusterMaintenanceMode, TestStateTransitionTimeout, TestInstanceOperation.

  • mvn test result: TestClusterMaintenanceMode and TestStateTransitionTimeout were validated locally under a CPU-throttle reproduction harness — both pass 3/3 iterations at the load that previously reproduced the flake, and pass at normal speed with no regression. TestInstanceOperation compiles cleanly but is not locally reproducible in isolation (deep dependsOnMethods chain); relying on CI for that class.

Changes that Break Backward Compatibility (Optional)

None. Test-only change; no production code, public API, config, or serialization-format changes.

Documentation (Optional)

None.

Commits

  • Commit subject uses the imperative mood; the body explains what and why (not how) and is wrapped.

Code Quality

  • Diff follows the helix-style formatting conventions.

…ation tests

Several integration tests poll for cluster convergence and then assert on
the outcome, but the polls use tight/default timeouts or wait on a different
condition than the one asserted. Under CI scheduling contention a poll can
return before the cluster converges, producing intermittent
`expected:<true> but was:<false>` failures unrelated to the code under test.

- TestClusterMaintenanceMode: re-point 13 polls at the exact asserted
  condition (maintenance-signal state, specific live-instance names, or the
  history entry's OPERATION_TYPE), assert the poll result instead of ignoring
  it, and wait TestHelper.WAIT_DURATION.
- TestStateTransitionTimeout: assert the verifyByPolling result and use the
  60s overload instead of the default 30s.
- TestInstanceOperation: widen five 30s verifier polls to
  TestHelper.WAIT_DURATION (fixes the testEvacuateWithDisabledPartition flake
  seen in CI).

Test-only change; no production code is modified. Widening is safe because a
poll returns as soon as its predicate is true, so passing runs are unaffected
and only slow-but-correct runs get the extra time they need.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant