Skip to content

Separate init and runtime timeout budgets, add per-request timeouts#1953

Open
moguchev wants to merge 1 commit into
apache:trunkfrom
moguchev:issue_1919
Open

Separate init and runtime timeout budgets, add per-request timeouts#1953
moguchev wants to merge 1 commit into
apache:trunkfrom
moguchev:issue_1919

Conversation

@moguchev

@moguchev moguchev commented May 20, 2026

Copy link
Copy Markdown

Related Issue: #1919

Splits previously conflated timeout budgets so a low user-supplied Timeout no longer breaks session establishment, and adds dedicated client-side budgets for system queries and per-request overrides. Also fixes a potential controlConn reconnect deadlock.

  • ClusterConfig.ReadTimeout: socket-level read deadline, defaults to 11s, falls back to ConnectTimeout if unset. Decoupled from Timeout, which is now purely a per-request client-side budget.
  • ClusterConfig.Metadata.SystemRequestTimeout (default 60s): per-request budget for system/system_schema queries and other runtime control-conn frames. Applied via Conn.querySystem, controlConn.query and controlConn.writeFrame.
  • Query/Batch.WithRequestTimeout / SetRequestTimeout / GetRequestTimeout: per-request override of ClusterConfig.Timeout, enforced via callReq.timer in Conn.exec independently of the socket deadline. 0 disables the client-side timer.
  • Conn.finalizeConnection: switches an initialized connection from ConnectTimeout-bound init mode to operational ReadTimeout / WriteTimeout / Metadata.SystemRequestTimeout. Called by hostConnPool.connect, controlConn.connect and controlConn.attemptReconnectToAnyOfHosts. Until then, all socket reads/writes and internal init-phase requests (STARTUP/AUTH/OPTIONS/system.local/USE keyspace) are pinned to ConnectTimeout. Fixes "no connections were made when creating the session" when Timeout is low.
  • controlConn.HandleError now launches reconnect in a goroutine to avoid a potential deadlock during a connection storm where attemptReconnect holds the controlConn mutex while doing DNS resolution / TCP dial.
  • connReader.timeout, deadlineContextWriter.timeout and writeCoalescer.timeout are now atomics to allow finalizeConnection to update them concurrently with reader/writer goroutines.
  • Regression tests: TestNewConnectWithLowTimeout, TestInitSchemaQueryRespectsConnectTimeout, TestControlConnQueryRespectsSystemRequestTimeout, TestQueryRequestTimeout, TestControlConn_HandleError_LaunchesReconnectInGoroutine.
  • README: new "Connection Timeouts" section documenting the layers and picking guidance.

Splits previously conflated timeout budgets so a low user-supplied Timeout
no longer breaks session establishment, and adds dedicated client-side
budgets for system queries and per-request overrides. Also fixes a
potential controlConn reconnect deadlock.

* ClusterConfig.ReadTimeout: socket-level read deadline, defaults to 11s,
  falls back to ConnectTimeout if unset. Decoupled from Timeout, which is
  now purely a per-request client-side budget.
* ClusterConfig.Metadata.SystemRequestTimeout (default 60s): per-request
  budget for system/system_schema queries and other runtime control-conn
  frames. Applied via Conn.querySystem, controlConn.query and
  controlConn.writeFrame.
* Query/Batch.WithRequestTimeout / SetRequestTimeout / GetRequestTimeout:
  per-request override of ClusterConfig.Timeout, enforced via
  callReq.timer in Conn.exec independently of the socket deadline. 0
  disables the client-side timer.
* Conn.finalizeConnection: switches an initialized connection from
  ConnectTimeout-bound init mode to operational ReadTimeout / WriteTimeout
  / Metadata.SystemRequestTimeout. Called by hostConnPool.connect,
  controlConn.connect and controlConn.attemptReconnectToAnyOfHosts.
  Until then, all socket reads/writes and internal init-phase requests
  (STARTUP/AUTH/OPTIONS/system.local/USE keyspace) are pinned to
  ConnectTimeout. Fixes "no connections were made when creating the
  session" when Timeout is low.
* controlConn.HandleError now launches reconnect in a goroutine to avoid
  a potential deadlock during a connection storm where attemptReconnect
  holds the controlConn mutex while doing DNS resolution / TCP dial.
* connReader.timeout, deadlineContextWriter.timeout and
  writeCoalescer.timeout are now atomics to allow finalizeConnection to
  update them concurrently with reader/writer goroutines.
* Regression tests: TestNewConnectWithLowTimeout,
  TestInitSchemaQueryRespectsConnectTimeout,
  TestControlConnQueryRespectsSystemRequestTimeout, TestQueryRequestTimeout,
  TestControlConn_HandleError_LaunchesReconnectInGoroutine.
* README: new "Connection Timeouts" section documenting the layers and
  picking guidance.
@joao-r-reis

joao-r-reis commented May 20, 2026

Copy link
Copy Markdown
Contributor

We want to prioritize a fix for a quick patch release so we need a PR that focuses on removing the connection read deadline and therefore making cfg.Timeout apply to just the request timeout (callReq timer).

After that we can work on a more comprehensive improvement of the existing timeout configuration fields such as this PR.

cc @worryg0d

@moguchev

moguchev commented May 20, 2026

Copy link
Copy Markdown
Author

We want to prioritize a fix for a quick patch release so we need a PR that focuses on removing the connection read deadline and therefore making cfg.Timeout apply to just the request timeout (callReq timer).

After that we can work on a more comprehensive improvement of the existing timeout configuration fields such as this PR.

Of course, this PR is more for inspiration for the future works with timeouts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants