Skip to content

feat: Control client-side timeout and total waiting time for model responses #24016

@sehoon38

Description

@sehoon38

What would you like to be added?

Provide a simple way to control client-side timeouts for server requests made by the CLI (e.g., generateContentStream, generateContent, and any other backend API calls).
Introduce a Mendel (experiment) flag to control these timeouts globally.
Additionally, control the total waiting time for a server response, including retries. If the cumulative duration exceeds a defined threshold (e.g., thresholdSec), the system should stop retrying and cancel the operation.

Why is this needed?

Currently, timeouts are not globally configurable for all server requests, and retries can potentially continue for a long time without a cap on the total duration. Providing these controls allows for better resource management and prevents sessions from hanging indefinitely in suboptimal network conditions or during server unavailability.

Additional context?

The Mendel flag should allow for experimentation and gradual rollout of these global timeout settings.
The cumulative duration check should be integrated into the core retry logic (e.g., retryWithBackoff in packages/core/src/utils/retry.ts) or the base client configuration so that it applies uniformly to all network requests.

Metadata

Metadata

Assignees

Labels

area/platformIssues related to Build infra, Release mgmt, Testing, Eval infra, Capacity, Quota mgmtworkstream-rollupLabel used to tag epics and features that are associated with one of the three primary workstreams🔒 maintainer only⛔ Do not contribute. Internal roadmap item.

Type

No fields configured for Task.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions