You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
CPU-based model inference (`vllm-cpu`) generates tokens at ~0.3–0.4 tokens/s. Responses requiring more than ~24 tokens exceed the default 60s upstream timeout enforced by ingress-nginx and APISIX.
23
+
CPU-based model inference (`vllm-cpu`) generates tokens at ~0.3-0.4 tokens/s. Responses requiring more than ~24 tokens exceed the default 60s upstream timeout enforced by ingress-nginx and APISIX.
24
24
25
25
**Fix:**
26
26
@@ -75,5 +75,5 @@ Re-run the inference request and confirm a `200 OK` response is returned within
75
75
**Notes:**
76
76
77
77
- The nginx ingress annotation takes effect immediately; no pod restart required.
78
-
- For GPU-based deployments this timeout is rarely needed as throughput is significantly higher (30–50 tokens/s vs 0.3–0.4 tokens/s on CPU).
78
+
- For GPU-based deployments this timeout is rarely needed as throughput is significantly higher (30-50 tokens/s vs 0.3-0.4 tokens/s on CPU).
79
79
- If requests still time out after increasing both timeouts, reduce `max_tokens` in the request payload to limit response length.
0 commit comments