Skip to content

Commit cf3538f

Browse files
committed
Fix Dataflow legacy worker abort loop thread death issue
Previously, when the service asked the worker to abort, it threw ReadLoopAbortedException, which extended InterruptedException. MapTaskExecutor caught this and, in an attempt to preserve the interrupted status, set the interrupted bit on the thread. However, since this was a logical abort and not a real thread interrupt, setting the interrupted bit caused subsequent operations on the thread (like the backoff sleep in DataflowBatchWorkerHarness) to immediately fail with InterruptedException, leading to all worker threads dying and the harness hanging. This fix: 1. Changes ReadLoopAbortedException to extend InterruptedIOException instead of InterruptedException, as it is a logical I/O abort and should not trigger thread interrupt handling. 2. Hardens DataflowBatchWorkerHarness to throw a RuntimeException if all worker threads die, ensuring the JVM exits with a non-zero code so the runner can restart it. Ref: b/512366613
1 parent be50185 commit cf3538f

2 files changed

Lines changed: 3 additions & 1 deletion

File tree

runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowBatchWorkerHarness.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,7 @@ public void run() throws InterruptedException {
9292

9393
worker.startStatusServer();
9494
processWork(pipelineOptions, worker, Sleeper.DEFAULT);
95+
throw new RuntimeException("All worker threads have died. Exiting.");
9596
}
9697

9798
// ExponentialBackOff parameters for the task retry strategy.

runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperation.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
import java.io.Closeable;
2121
import java.io.IOException;
22+
import java.io.InterruptedIOException;
2223
import java.util.Observable;
2324
import java.util.Observer;
2425
import java.util.concurrent.CancellationException;
@@ -47,7 +48,7 @@
4748
"nullness" // TODO(https://github.com/apache/beam/issues/20497)
4849
})
4950
public class ReadOperation extends Operation {
50-
public static class ReadLoopAbortedException extends InterruptedException {
51+
public static class ReadLoopAbortedException extends InterruptedIOException {
5152
public ReadLoopAbortedException() {
5253
super("Read loop was aborted.");
5354
}

0 commit comments

Comments
 (0)