Skip to content

docs: Warn operators to bound grpc-max-response-pool-size #8880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yinggeh merged 1 commit into from
Jul 13, 2026
+7 −3
Merged

docs: Warn operators to bound grpc-max-response-pool-size #8880

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions docs/customization_guide/deploy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,11 @@ Specifies the maximum number of states (inference request/response queues) that
Specifies the maximum number of inference response objects that can remain allocated in each response queue at any given time. This option is particularly useful in decoupled mode, where multiple responses are generated for a single request. By default, this value is set to `INT_MAX`.

> [!Warning]
> Setting this value too low may negatively impact performance.


> Operators exposing decoupled gRPC streaming to untrusted clients should
> explicitly configure `--grpc-max-response-pool-size` to a bounded value
> appropriate for their workload, rather than relying on the default
> `INT_MAX`. Choose a value based on model behavior (decoupled vs
> non-decoupled), expected concurrency, and available memory.

> [!Warning]
> Setting this value too low may negatively impact performance.
Loading