Skip to content

Commit 1d52bc5

Browse files
author
Harika
committed
update troubleshooting.md
1 parent 3410c1d commit 1d52bc5

2 files changed

Lines changed: 3 additions & 76 deletions

File tree

third_party/Dell/model-deployment/llama-3.1-8b-instruct/xeon-deployment.md

Lines changed: 0 additions & 73 deletions
This file was deleted.

third_party/Dell/model-deployment/troubleshooting.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ This section provides common issues observed when running inference against mode
99

1010
### 1. Gateway Timeout (504) on Inference Requests
1111

12-
**Context:** Model deployed via Helm commands. Inference request sent through the ingress stack (ingress-nginx APISIX vLLM service).
12+
**Context:** Model deployed via Helm commands. Inference request sent through the ingress stack (ingress-nginx -> APISIX -> vLLM service).
1313

1414
**Error:** Inference requests return `504 Gateway Timeout` after 60 seconds:
1515

@@ -20,7 +20,7 @@ upstream timed out (110: Operation timed out) ... 60.001
2020

2121
**Cause:**
2222

23-
CPU-based model inference (`vllm-cpu`) generates tokens at ~0.30.4 tokens/s. Responses requiring more than ~24 tokens exceed the default 60s upstream timeout enforced by ingress-nginx and APISIX.
23+
CPU-based model inference (`vllm-cpu`) generates tokens at ~0.3-0.4 tokens/s. Responses requiring more than ~24 tokens exceed the default 60s upstream timeout enforced by ingress-nginx and APISIX.
2424

2525
**Fix:**
2626

@@ -75,5 +75,5 @@ Re-run the inference request and confirm a `200 OK` response is returned within
7575
**Notes:**
7676

7777
- The nginx ingress annotation takes effect immediately; no pod restart required.
78-
- For GPU-based deployments this timeout is rarely needed as throughput is significantly higher (3050 tokens/s vs 0.30.4 tokens/s on CPU).
78+
- For GPU-based deployments this timeout is rarely needed as throughput is significantly higher (30-50 tokens/s vs 0.3-0.4 tokens/s on CPU).
7979
- If requests still time out after increasing both timeouts, reduce `max_tokens` in the request payload to limit response length.

0 commit comments

Comments
 (0)