[Doc] Add V100 (CC 7.0) legacy GPU deployment tutorial#973
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive tutorial and Helm configuration templates for deploying vLLM on legacy Tesla V100 (Volta, CC 7.0) GPUs, covering single-GPU and multi-GPU setups. The review feedback correctly identifies a consistent factual error across the documentation and configuration comments regarding the default shared memory size (shmSize). While the text claims the chart's default is too small and needs to be bumped to 4Gi, the chart's default is actually 20Gi. The feedback suggests clarifying that the Kubernetes default (64Mi) is the bottleneck, and that 4Gi is proposed as a resource-optimized value.
| In addition to the single-GPU requirements, multi-GPU adds the following: | ||
|
|
||
| - **`tensorParallelSize` must live in `vllmConfig`, NOT in `extraArgs`.** This is the easiest mistake to make. The chart renders `vllmConfig.tensorParallelSize` into the `--tensor-parallel-size` flag for you. If you instead add `--tensor-parallel-size` (or `--tensor_parallel_size`) to `extraArgs`, the value is **silently ignored**, tensor parallelism stays at 1, and the pod then OOMs while loading the model. Always set it in `vllmConfig`. | ||
| - **`shmSize: "4Gi"`** — tensor-parallel all-reduce uses NCCL, which communicates between the per-GPU worker processes through `/dev/shm`. The chart's default shared-memory size is too small for multi-GPU NCCL and causes init hangs or bus errors. `requestGPU > 1` requires bumping `shmSize`. |
There was a problem hiding this comment.
The explanation states that the chart's default shared-memory size is too small and requires bumping. However, in helm/values.yaml (line 198), the default shmSize is actually set to "20Gi", which is significantly larger than "4Gi".
Therefore, setting shmSize: "4Gi" actually reduces the shared memory size from the chart's default (though 4Gi is still sufficient and more resource-friendly than 20Gi). The documentation should be updated to clarify that the Kubernetes default (64Mi) is too small, but the chart's default (20Gi) is already sufficient, and 4Gi is an optimized/reduced value.
| - **`shmSize: "4Gi"`** — tensor-parallel all-reduce uses NCCL, which communicates between the per-GPU worker processes through `/dev/shm`. The chart's default shared-memory size is too small for multi-GPU NCCL and causes init hangs or bus errors. `requestGPU > 1` requires bumping `shmSize`. | |
| - **shmSize: "4Gi"** — tensor-parallel all-reduce uses NCCL, which communicates between the per-GPU worker processes through /dev/shm. The Kubernetes default shared-memory size (64Mi) is too small for multi-GPU NCCL and causes init hangs or bus errors. While the chart's default is 20Gi, we set it to 4Gi here as a resource-friendly value that is still more than sufficient. |
| # shmSize backs the /dev/shm that NCCL uses for inter-GPU shared-memory | ||
| # transport during tensor-parallel all-reduce. The chart default is too | ||
| # small for multi-GPU NCCL and causes init hangs / bus errors. |
There was a problem hiding this comment.
The comment states that the chart's default shared-memory size is too small. However, the chart's default shmSize is actually "20Gi" (see helm/values.yaml line 198), which is much larger than "4Gi". The Kubernetes default (64Mi) is what is too small. Let's update the comment to be accurate.
# shmSize backs the /dev/shm that NCCL uses for inter-GPU shared-memory
# transport during tensor-parallel all-reduce. The Kubernetes default (64Mi)
# is too small and causes bus errors, but the chart's default (20Gi) is plenty.
# We set it to 4Gi here as a resource-friendly sufficient value.| - name: VLLM_ENABLE_CUDA_COMPATIBILITY | ||
| value: "1" | ||
|
|
||
| # Larger /dev/shm for 4-way NCCL shared-memory transport. |
There was a problem hiding this comment.
The comment mentions "Larger /dev/shm", but 4Gi is actually smaller than the chart's default of 20Gi. It is only larger compared to the Kubernetes default of 64Mi. Let's clarify this in the comment.
# Set /dev/shm size for 4-way NCCL shared-memory transport.
# Note: The Kubernetes default (64Mi) is too small, but the chart's default (20Gi) is plenty.
# We set it to 4Gi here as a resource-friendly sufficient value.| | Pod scheduled on the wrong (non-V100) node | Used `nodeSelector`, which this chart does not render. | Use `nodeSelectorTerms` (node affinity) instead. | | ||
| | Both old and new pods stuck `Pending` after a redeploy | Default `RollingUpdate` starts the new pod before the old one frees its GPUs — GPU deadlock. | Set `servingEngineSpec.strategy.type: Recreate`. | | ||
| | Engine reports `tensor_parallel_size=1` even though you set it | `--tensor-parallel-size` was placed in `extraArgs`, where it is silently ignored. | Set `tensorParallelSize` inside `vllmConfig`. | | ||
| | NCCL init hang or bus error on multi-GPU startup | `/dev/shm` too small for NCCL shared-memory transport. | Set `shmSize: "4Gi"` (required when `requestGPU > 1`). | |
There was a problem hiding this comment.
Since the chart's default is 20Gi, setting shmSize: "4Gi" is not strictly required to avoid the error (as the default is already sufficient). Let's update the table row to reflect this.
| | NCCL init hang or bus error on multi-GPU startup | `/dev/shm` too small for NCCL shared-memory transport. | Set `shmSize: "4Gi"` (required when `requestGPU > 1`). | | |
| | NCCL init hang or bus error on multi-GPU startup | /dev/shm too small for NCCL shared-memory transport. | Set shmSize: "4Gi" (or rely on the chart's default 20Gi). | |
Signed-off-by: AmitChaubey <amit.katyayana@gmail.com>
amit-chaubey
force-pushed
the
tutorial/legacy-gpu-v100-cc70-deployment
branch
from
2ec1d67 to
dccde21
Compare
|
thanks @ruizhang0101 |
2 participants
What
Adds a tutorial for deploying on Volta GPUs (Tesla V100, CC 7.0), which
vllm/vllm-openai:latest(v0.9+) no longer supports.Covers the required image pin (
v0.8.5) and flags (dtype: half,v0: "1",--enforce-eager,VLLM_ENABLE_CUDA_COMPATIBILITY), plusmulti-GPU TP=2/TP=4 (
tensorParallelSize,shmSize,Recreate,nodeSelectorTerms) and a common-errors table.Added
tutorials/25-v100-legacy-gpu-deployment.mdtutorials/assets/values-25-v100-single-gpu.yaml(Qwen2.5-0.5B, TP=1)tutorials/assets/values-25-v100-multi-gpu-tp2.yaml(Qwen2.5-7B, TP=2)tutorials/assets/values-25-v100-multi-gpu-tp4.yaml(Qwen2.5-14B, TP=4)Validated on OKD 4.19 / k8s v1.32.8, V100 SXM2 16GB, vllm-stack-0.1.11.
Values files are helm-lint clean.
-swhen doinggit commit[Bugfix],[Feat], and[CI].Details
... NA...