You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: remove unverified timeout scenarios and NLB troubleshooting from Spark connector
Remove the Common timeout scenarios section and the Connection resets on
AWS PrivateLink or NLB troubleshooting entry. The NLB broken pipe scenario
could not be reproduced from EMR against the staging endpoint (public HTTPS
path, no NLB in the TCP path), so this content is not verified and should
not be published.
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/apache-spark/spark-native-connector.md
+1-47Lines changed: 1 addition & 47 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1847,7 +1847,7 @@ The connector enforces its own timeouts that are independent of any Spark or Cli
1847
1847
1848
1848
The 60-second query cap is enforced by the connector regardless of `max_execution_time` or any other server setting. If a read query takes longer than 60 seconds end-to-end, the connector will abort it. There is no `spark.clickhouse.*` setting to override this value.
1849
1849
1850
-
Inserts have no connector-level timeout. This means a stalled or very slow insert will hang until a network device terminates the connection — which can produce a **"Broken pipe"** error (see [Connection resets on AWS PrivateLink or NLB](#troubleshooting-connection-resets) below).
1850
+
Inserts have no connector-level timeout. This means a stalled or very slow insert will hang until a network device terminates the connection — which can produce a **"Broken pipe"** error.
@@ -1869,27 +1869,6 @@ These are ClickHouse query settings sent with each request. They instruct the Cl
1869
1869
|`option.clickhouse_setting_max_execution_time`|`0` (unlimited) | seconds | Server-side hard cap on query execution time. Useful for preventing runaway reads from consuming server resources, but **does not override the connector's 60-second query timeout**. |
1870
1870
|`option.clickhouse_setting_session_timeout`|`60`| seconds | HTTP session lifetime on the server. |
1871
1871
1872
-
### Common timeout scenarios {#timeout-scenarios}
1873
-
1874
-
**Long-running reads (>60 seconds)**
1875
-
1876
-
The connector's hard-coded 60-second query timeout will abort reads that take longer, even if the server could complete them. There is no configuration knob to extend or disable this cap. The only workaround is to reduce the amount of data returned by a single query so each read completes within 60 seconds:
1877
-
1878
-
- Add `WHERE` predicates to filter rows before they reach Spark.
1879
-
- Use partition pruning so the connector issues one query per partition rather than one large scan.
1880
-
1881
-
**Insert "Broken pipe" or "Connection reset" errors**
1882
-
1883
-
Typically caused by a network appliance (AWS NLB, PrivateLink, corporate proxy) silently dropping idle connections. The connector's connection pool holds connections open between operations; if the appliance's idle timeout (350 seconds for AWS NLB) expires, the connection is dropped at the network level without notification. The connector then tries to reuse a dead connection and gets a broken pipe.
Setting `connection_ttl` below the appliance's idle timeout forces the connector to retire connections proactively. See [Connection resets on AWS PrivateLink or NLB](#troubleshooting-connection-resets) for full details.
### Connection resets on AWS PrivateLink or NLB {#troubleshooting-connection-resets}
1971
-
1972
-
**Symptom**: Reads or writes intermittently fail with `Broken pipe`, `Connection reset by peer`, or `java.io.IOException: Connection closed` — particularly on longer-running jobs or after a period of inactivity.
1973
-
1974
-
**Cause**: AWS Network Load Balancers (NLB) and PrivateLink silently drop TCP connections that have been idle for **350 seconds**. The connector's connection pool keeps HTTP connections alive between operations. When the NLB drops a pooled connection, neither the connector nor ClickHouse is notified. The next operation that picks up that connection fails immediately.
1975
-
1976
-
Inserts are especially vulnerable: there is no connector-level insert timeout, so a slow or large insert can stall mid-flight when the underlying connection is cut.
1977
-
1978
-
**Fix**: Retire pooled connections before the NLB does by setting `connection_ttl` below 350 seconds. Also set `socket_timeout` to bound individual operations and disable HTTP keep-alive to prevent stale connections from persisting in the pool:
For very large inserts that are genuinely expected to take several minutes, switch to async inserts so the server acknowledges receipt quickly and the connector is not waiting on a long-lived open connection:
### KEEPER_EXCEPTION during writes {#troubleshooting-keeper-exception}
1996
1950
1997
1951
**Symptom**: Writes fail with `Coordination::Exception: Can't get data for node ... node doesn't exist (KEEPER_EXCEPTION)`. Retries may produce duplicate data.
0 commit comments