What happened?
What happened:
After the client-v2 migration (Beam 2.72.0), ClickHouseIO.WriteFn builds com.clickhouse.client.api.Client without a socket timeout, connect timeout, or TCP keepalive, and calls client.insert(...).get() (blocking, no timeout) in flush(). When a connection is silently dropped (LB idle timeout, server restart, half-open TCP), the worker blocks forever in SSLSocketInputRecord.read while awaiting the response header.
On Dataflow this manifests as Operation ongoing in bundle indefinitely without outputting or completing, stack ending at ClickHouseIO.java:603 (flush) → :562 (finishBundle). The worker never recovers.
Critically, withMaxRetries/withInitialBackoff provide no protection: the retry loop only catches exceptions, and a hung socket never throws.
Operation ongoing in bundle process_bundle-3299-5293 for PTransform{id=Write [REDACTED] to ClickHouse/ParDo(AutoValue_ClickHouseIO_Write)/ParMultiDo(AutoValue_ClickHouseIO_Write)-ptransform-92, name=Write [REDACTED] to ClickHouse/ParDo(AutoValue_ClickHouseIO_Write)/ParMultiDo(AutoValue_ClickHouseIO_Write)-ptransform-92, state=finish} for at least 4h06m18s without outputting or completing:\nat java.base/sun.nio.ch.Net.poll(Native Method)\n at java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:191)\n at java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:201)\n at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)\n at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)\n at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)\n at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099)\n at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)\n at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)\n at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)\n at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461)\n at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:250)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:56)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:331)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:196)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.lambda$execute$0(InternalExecRuntime.java:236)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:791)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:233)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:120)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:198)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:110)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:183)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55)\n at com.clickhouse.shaded.org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183)\n at com.clickhouse.client.api.internal.HttpAPIClientHelper.executeRequest(HttpAPIClientHelper.java:453)\n at com.clickhouse.client.api.Client.lambda$insert$3(Client.java:1474)\n at com.clickhouse.client.api.Client.runAsyncOperation(Client.java:2012)\n at com.clickhouse.client.api.Client.insert(Client.java:1518)\n at com.clickhouse.client.api.Client.insert(Client.java:1376)\n at com.clickhouse.client.api.Client.insert(Client.java:1320)\n at org.apache.beam.sdk.io.clickhouse.ClickHouseIO$WriteFn.flush(ClickHouseIO.java:603)\n at org.apache.beam.sdk.io.clickhouse.ClickHouseIO$WriteFn.finishBundle(ClickHouseIO.java:562)\n at org.apache.beam.sdk.io.clickhouse.AutoValue_ClickHouseIO_WriteFn$DoFnInvoker$a1edf190.invokeFinishBundle(Unknown Source)\n at org.apache.beam.fn.harness.FnApiDoFnRunner.finishBundle(FnApiDoFnRunner.java:1303)\n at org.apache.beam.fn.harness.data.PTransformFunctionRegistry.lambda$register$0(PTransformFunctionRegistry.java:115)\n at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:561)\n at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)\n at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)\n at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)\n at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)\n at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n at java.base/java.lang.Thread.run(Thread.java:1583)
Expected: A stalled connection should fail with a (retryable) timeout, not hang indefinitely.
Versions: Beam 2.72.0, ClickHouse client-v2, DataflowRunner, Java 21.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
What happened:
After the client-v2 migration (Beam 2.72.0), ClickHouseIO.WriteFn builds com.clickhouse.client.api.Client without a socket timeout, connect timeout, or TCP keepalive, and calls client.insert(...).get() (blocking, no timeout) in flush(). When a connection is silently dropped (LB idle timeout, server restart, half-open TCP), the worker blocks forever in SSLSocketInputRecord.read while awaiting the response header.
On Dataflow this manifests as Operation ongoing in bundle indefinitely without outputting or completing, stack ending at ClickHouseIO.java:603 (flush) → :562 (finishBundle). The worker never recovers.
Critically, withMaxRetries/withInitialBackoff provide no protection: the retry loop only catches exceptions, and a hung socket never throws.
Operation ongoing in bundle process_bundle-3299-5293 for PTransform{id=Write [REDACTED] to ClickHouse/ParDo(AutoValue_ClickHouseIO_Write)/ParMultiDo(AutoValue_ClickHouseIO_Write)-ptransform-92, name=Write [REDACTED] to ClickHouse/ParDo(AutoValue_ClickHouseIO_Write)/ParMultiDo(AutoValue_ClickHouseIO_Write)-ptransform-92, state=finish} for at least 4h06m18s without outputting or completing:\nat java.base/sun.nio.ch.Net.poll(Native Method)\n at java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:191)\n at java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:201)\n at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)\n at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)\n at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)\n at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099)\n at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)\n at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)\n at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)\n at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461)\n at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:250)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:56)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:331)\n at com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:196)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.lambda$execute$0(InternalExecRuntime.java:236)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:791)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:233)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:120)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:198)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:110)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:183)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)\n at com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55)\n at com.clickhouse.shaded.org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183)\n at com.clickhouse.client.api.internal.HttpAPIClientHelper.executeRequest(HttpAPIClientHelper.java:453)\n at com.clickhouse.client.api.Client.lambda$insert$3(Client.java:1474)\n at com.clickhouse.client.api.Client.runAsyncOperation(Client.java:2012)\n at com.clickhouse.client.api.Client.insert(Client.java:1518)\n at com.clickhouse.client.api.Client.insert(Client.java:1376)\n at com.clickhouse.client.api.Client.insert(Client.java:1320)\n at org.apache.beam.sdk.io.clickhouse.ClickHouseIO$WriteFn.flush(ClickHouseIO.java:603)\n at org.apache.beam.sdk.io.clickhouse.ClickHouseIO$WriteFn.finishBundle(ClickHouseIO.java:562)\n at org.apache.beam.sdk.io.clickhouse.AutoValue_ClickHouseIO_WriteFn$DoFnInvoker$a1edf190.invokeFinishBundle(Unknown Source)\n at org.apache.beam.fn.harness.FnApiDoFnRunner.finishBundle(FnApiDoFnRunner.java:1303)\n at org.apache.beam.fn.harness.data.PTransformFunctionRegistry.lambda$register$0(PTransformFunctionRegistry.java:115)\n at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:561)\n at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)\n at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)\n at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)\n at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)\n at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n at java.base/java.lang.Thread.run(Thread.java:1583)Expected: A stalled connection should fail with a (retryable) timeout, not hang indefinitely.
Versions: Beam 2.72.0, ClickHouse client-v2, DataflowRunner, Java 21.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components