[AUTOCUT] Gradle Check Flaky Test Report for WarmIndexSegmentReplicationIT

[opensearch-ci-bot]

Flaky Test Report for WarmIndexSegmentReplicationIT

Noticed the WarmIndexSegmentReplicationIT has some flaky, failing tests that failed during post-merge actions and timer trigerred run on main branch.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
main	null	66915 67982 74319 74387 25 57203 60336 60346 60883 61065 61090 62120 62599 63244 63374 63786 64498 65023 66136 66617 66899 66989 67400 67939 68254 69312 69319 69344 69395 69498 69671 69725 69763 69841 69881 69939 69943 70009 70013 70081 70347 70473 70475 70694 70810 70846 71342 71359 71567 72259 72279 72466 72567 72997 73194 73321 73668 73920 73945 74046 74264 74298 74404 74598 75045 75105 75112 75130 75502	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.classMethod org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testIndexWhileRecoveringReplica org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicationAfterForceMergeOnPrimaryShardsOnly org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicationAfterPrimaryRefreshAndFlush org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testDropPrimaryDuringReplication org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testIndexReopenClose org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testNodeDropWithOngoingReplication org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicationAfterForceMerge org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testScrollCreatedOnReplica org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testCancelPrimaryAllocation org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicaHasDiffFilesThanPrimary org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testCancellation org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryReceivesDocsDuringReplicaRecovery org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRealtimeTermVectorRequestsSuccessful org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicaAlreadyAtCheckpoint org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testDeleteOperations org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRealtimeGetRequestsSuccessful org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRealtimeMultiGetRequestsUnsuccessful org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimaryAndReplicaWithDocuments org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary_NoReplicas org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testScrollWithConcurrentIndexAndSearch org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testUpdateOperations
6f3f367	21332	75437	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary_NoReplicas
ac0f565	21425	75424	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicaHasDiffFilesThanPrimary
78d493d	21221	74815	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.classMethod org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted
bf943fa	21236	74804	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testCancellation
e8c539b	20930	73690	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicaHasDiffFilesThanPrimary
0e2783c	20842	72436	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testIndexWhileRecoveringReplica
76c7b66	20695	71615	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary_NoReplicas
5041d43	20361	70392	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted
a7c85f2	20427	70308	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testScrollCreatedOnReplica
9e586e5	20331	69543	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testCancelPrimaryAllocation
ce178f6	20184	68585	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testNodeDropWithOngoingReplication
859f7c3	20132	68098	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRealtimeTermVectorRequestsSuccessful
99f3189	19495	64883	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testRestartPrimary
4b9aa27	18550	63886	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testReplicationAfterPrimaryRefreshAndFlush
c56da68	19296	63885	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate
2cec4b5	18646	59878	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testScrollWithConcurrentIndexAndSearch
1adc8cd	18585	59573	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate
cb30055	18060	57402	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted
0795bb2	18099	57396	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryStopped_ReplicaPromoted

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the WarmIndexSegmentReplicationIT class are:

• 17617
• 21311
• 18147
• 18333
• 17507
• 17902
• 19003
• 18467
• 18565
• 18599
• 19703
• 20160
• 20192
• 20503
• 21038
• 17556
• 17580
• 17598
• 17606
• 17792
• 17890
• 18069
• 18070
• 18116
• 18166
• 18195
• 18424
• 18573
• 18593
• 18610
• 18621
• 18631
• 18656
• 18664
• 18702
• 18720
• 18749
• 18800
• 18824
• 18826
• 18842
• 18887
• 18890
• 18929
• 18944
• 18951
• 18973
• 19001
• 19007
• 19054
• 19055
• 19132
• 19152
• 19165
• 19226
• 19278
• 19294
• 19307
• 19387
• 19508
• 19555
• 19576
• 19619
• 19648
• 19684
• 19711
• 19744
• 19782
• 19798
• 19857
• 19868
• 19933
• 19935
• 19966
• 20017
• 20057
• 20065
• 20080
• 20082
• 20123
• 20145
• 20204
• 20251
• 20280
• 20345
• 20351
• 20371
• 20377
• 20434
• 20478
• 20511
• 20514
• 20585
• 20623
• 20672
• 20683
• 20691
• 20702
• 20802
• 20810
• 20840
• 20854
• 20869
• 20924
• 20929
• 20978
• 21017
• 21048
• 21113
• 21145
• 21183
• 21202
• 21220
• 21290
• 21291
• 21305
• 21370
• 21418

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[andrross]

FYI @skumawat2025

[andrross]

https://build.ci.opensearch.org/job/gradle-check/57471/consoleText:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryReceivesDocsDuringReplicaRecovery" -Dtests.seed=C5CF2E73AD7D00AF -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-PR -Dtests.timezone=Europe/Busingen -Druntime.java=21

WarmIndexSegmentReplicationIT > testPrimaryReceivesDocsDuringReplicaRecovery FAILED
    java.lang.AssertionError: Expected search hits on node: node_t1 to be at least 2 but was: 1
        at __randomizedtesting.SeedInfo.seed([C5CF2E73AD7D00AF:68976B8B8CAA7ED1]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.indices.replication.SegmentReplicationBaseIT.lambda$waitForSearchableDocs$0(SegmentReplicationBaseIT.java:122)
        at org.opensearch.test.OpenSearchTestCase.assertBusyWithFixedSleepTime(OpenSearchTestCase.java:1214)
        at org.opensearch.indices.replication.SegmentReplicationBaseIT.waitForSearchableDocs(SegmentReplicationBaseIT.java:117)
        at org.opensearch.indices.replication.SegmentReplicationBaseIT.waitForSearchableDocs(SegmentReplicationBaseIT.java:112)
        at org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testPrimaryReceivesDocsDuringReplicaRecovery(WarmIndexSegmentReplicationIT.java:1211)

https://build.ci.opensearch.org/job/gradle-check/57472/consoleText:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testDeleteOperations" -Dtests.seed=3D049AF853D4BA75 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=el-CY -Dtests.timezone=Antarctica/Troll -Druntime.java=21

WarmIndexSegmentReplicationIT > testDeleteOperations FAILED
    java.lang.AssertionError: 
    Expected: a value equal to or greater than <2L>
         but: <0L> was less than <2L>
        at __randomizedtesting.SeedInfo.seed([3D049AF853D4BA75:8E50F0725EFA66E3]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.test.OpenSearchIntegTestCase.lambda$waitForDocs$3(OpenSearchIntegTestCase.java:1078)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1182)
        at org.opensearch.test.OpenSearchIntegTestCase.waitForDocs(OpenSearchIntegTestCase.java:1045)
        at org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testDeleteOperations(WarmIndexSegmentReplicationIT.java:794)

[skumawat2025]

Looks like some of the tests are still being flaky - Will take a look and get these fixed.