Skip to content

Commit b6c2e88

Browse files
committed
update troubleshooting.md
Signed-off-by: Harika <harika.devulapally@cloud2labs.com>
1 parent 971441a commit b6c2e88

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

third_party/Dell/model-deployment/troubleshooting.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ upstream timed out (110: Operation timed out) ... 60.001
2020

2121
**Cause:**
2222

23-
CPU-based model inference (`vllm-cpu`) generates tokens at ~0.3-0.4 tokens/s. Responses requiring more than ~24 tokens exceed the default 60s upstream timeout enforced by ingress-nginx and APISIX.
23+
Model inference using CPU-based configurations may encounter performance variances where processing times exceed the default 60-second upstream timeout enforced by ingress-nginx and APISIX, resulting in a timeout error.
2424

2525
**Fix:**
2626

@@ -75,5 +75,5 @@ Re-run the inference request and confirm a `200 OK` response is returned within
7575
**Notes:**
7676

7777
- The nginx ingress annotation takes effect immediately; no pod restart required.
78-
- For GPU-based deployments this timeout is rarely needed as throughput is significantly higher (30-50 tokens/s vs 0.3-0.4 tokens/s on CPU).
78+
- GPU-based deployments typically achieve significantly higher throughput, making this timeout adjustment rarely necessary compared to CPU configurations.
7979
- If requests still time out after increasing both timeouts, reduce `max_tokens` in the request payload to limit response length.

0 commit comments

Comments
 (0)