Skip to content

Commit de4ea8a

Browse files
committed
docs: remove unverified timeout scenarios and NLB troubleshooting from Spark connector
Remove the Common timeout scenarios section and the Connection resets on AWS PrivateLink or NLB troubleshooting entry. The NLB broken pipe scenario could not be reproduced from EMR against the staging endpoint (public HTTPS path, no NLB in the TCP path), so this content is not verified and should not be published.
1 parent 7aa14a8 commit de4ea8a

1 file changed

Lines changed: 1 addition & 47 deletions

File tree

docs/integrations/data-ingestion/apache-spark/spark-native-connector.md

Lines changed: 1 addition & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1847,7 +1847,7 @@ The connector enforces its own timeouts that are independent of any Spark or Cli
18471847

18481848
The 60-second query cap is enforced by the connector regardless of `max_execution_time` or any other server setting. If a read query takes longer than 60 seconds end-to-end, the connector will abort it. There is no `spark.clickhouse.*` setting to override this value.
18491849

1850-
Inserts have no connector-level timeout. This means a stalled or very slow insert will hang until a network device terminates the connection — which can produce a **"Broken pipe"** error (see [Connection resets on AWS PrivateLink or NLB](#troubleshooting-connection-resets) below).
1850+
Inserts have no connector-level timeout. This means a stalled or very slow insert will hang until a network device terminates the connection — which can produce a **"Broken pipe"** error.
18511851

18521852
### Java client connection timeouts {#timeout-java-client}
18531853

@@ -1869,27 +1869,6 @@ These are ClickHouse query settings sent with each request. They instruct the Cl
18691869
| `option.clickhouse_setting_max_execution_time` | `0` (unlimited) | seconds | Server-side hard cap on query execution time. Useful for preventing runaway reads from consuming server resources, but **does not override the connector's 60-second query timeout**. |
18701870
| `option.clickhouse_setting_session_timeout` | `60` | seconds | HTTP session lifetime on the server. |
18711871

1872-
### Common timeout scenarios {#timeout-scenarios}
1873-
1874-
**Long-running reads (>60 seconds)**
1875-
1876-
The connector's hard-coded 60-second query timeout will abort reads that take longer, even if the server could complete them. There is no configuration knob to extend or disable this cap. The only workaround is to reduce the amount of data returned by a single query so each read completes within 60 seconds:
1877-
1878-
- Add `WHERE` predicates to filter rows before they reach Spark.
1879-
- Use partition pruning so the connector issues one query per partition rather than one large scan.
1880-
1881-
**Insert "Broken pipe" or "Connection reset" errors**
1882-
1883-
Typically caused by a network appliance (AWS NLB, PrivateLink, corporate proxy) silently dropping idle connections. The connector's connection pool holds connections open between operations; if the appliance's idle timeout (350 seconds for AWS NLB) expires, the connection is dropped at the network level without notification. The connector then tries to reuse a dead connection and gets a broken pipe.
1884-
1885-
```text
1886-
spark.sql.catalog.clickhouse.option.connection_ttl 300000
1887-
spark.sql.catalog.clickhouse.option.socket_timeout 300000
1888-
spark.sql.catalog.clickhouse.option.http_keep_alive_timeout 0
1889-
```
1890-
1891-
Setting `connection_ttl` below the appliance's idle timeout forces the connector to retire connections proactively. See [Connection resets on AWS PrivateLink or NLB](#troubleshooting-connection-resets) for full details.
1892-
18931872
## Performance tuning {#performance-tuning}
18941873

18951874
### Read performance {#read-performance}
@@ -1967,31 +1946,6 @@ spark.catalog.refreshTable("clickhouse.database.table_name")
19671946

19681947
---
19691948

1970-
### Connection resets on AWS PrivateLink or NLB {#troubleshooting-connection-resets}
1971-
1972-
**Symptom**: Reads or writes intermittently fail with `Broken pipe`, `Connection reset by peer`, or `java.io.IOException: Connection closed` — particularly on longer-running jobs or after a period of inactivity.
1973-
1974-
**Cause**: AWS Network Load Balancers (NLB) and PrivateLink silently drop TCP connections that have been idle for **350 seconds**. The connector's connection pool keeps HTTP connections alive between operations. When the NLB drops a pooled connection, neither the connector nor ClickHouse is notified. The next operation that picks up that connection fails immediately.
1975-
1976-
Inserts are especially vulnerable: there is no connector-level insert timeout, so a slow or large insert can stall mid-flight when the underlying connection is cut.
1977-
1978-
**Fix**: Retire pooled connections before the NLB does by setting `connection_ttl` below 350 seconds. Also set `socket_timeout` to bound individual operations and disable HTTP keep-alive to prevent stale connections from persisting in the pool:
1979-
1980-
```text
1981-
spark.sql.catalog.clickhouse.option.connection_ttl 300000
1982-
spark.sql.catalog.clickhouse.option.socket_timeout 300000
1983-
spark.sql.catalog.clickhouse.option.http_keep_alive_timeout 0
1984-
```
1985-
1986-
For very large inserts that are genuinely expected to take several minutes, switch to async inserts so the server acknowledges receipt quickly and the connector is not waiting on a long-lived open connection:
1987-
1988-
```text
1989-
spark.sql.catalog.clickhouse.option.clickhouse_setting_async_insert 1
1990-
spark.sql.catalog.clickhouse.option.clickhouse_setting_wait_for_async_insert 1
1991-
```
1992-
1993-
---
1994-
19951949
### KEEPER_EXCEPTION during writes {#troubleshooting-keeper-exception}
19961950

19971951
**Symptom**: Writes fail with `Coordination::Exception: Can't get data for node ... node doesn't exist (KEEPER_EXCEPTION)`. Retries may produce duplicate data.

0 commit comments

Comments
 (0)