Skip to content

Commit c7c78e0

Browse files
committed
[MINOR][CORE][TESTS] Fix flaky DAGSchedulerSuite test for push-based shuffle
### What changes were proposed in this pull request? Add `sc.listenerBus.waitUntilEmpty()` before the `completedStage` assertion in the push-based shuffle test in `DAGSchedulerSuite`. ### Why are the changes needed? The test was intermittently failing due to a race condition. `SparkListenerStageCompleted` events are delivered asynchronously through the listener bus, so the `completedStage` assertion could execute before the event was processed, causing a spurious failure. Waiting for the listener bus to drain before asserting eliminates the race. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing test in `DAGSchedulerSuite`. The fix removes a race condition rather than adding new test logic. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 4.6 (Claude Code) Closes #55221 from dbtsai/dagscheduler-flaky-fix. Authored-by: DB Tsai <dbtsai@dbtsai.com> Signed-off-by: DB Tsai <dbtsai@dbtsai.com>
1 parent e8b5cb8 commit c7c78e0

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5827,6 +5827,7 @@ class DAGSchedulerSuite extends SparkFunSuite with TempLocalSparkContext with Ti
58275827
// Map stage completes successfully,
58285828
completeShuffleMapStageSuccessfully(0, 0, 3, Seq("hostA", "hostB"))
58295829
taskIdCount += 2
5830+
sc.listenerBus.waitUntilEmpty()
58305831
assert(completedStage === List(0))
58315832

58325833
// Now submit the first reducer stage

0 commit comments

Comments
 (0)