|
| 1 | +# Tutorial: Deploying on Tesla V100 and Other Legacy (Volta, CC 7.0) GPUs |
| 2 | + |
| 3 | +## Introduction |
| 4 | + |
| 5 | +This tutorial is for users who run the vLLM Production Stack on **Volta-generation NVIDIA GPUs** such as the Tesla V100 (CUDA Compute Capability 7.0). Newer prebuilt vLLM images (`vllm/vllm-openai:latest`, vLLM v0.9+) **dropped support for Compute Capability below 8.0**, so a V100 deployment that works on Ampere/Hopper out of the box will fail on Volta with errors like: |
| 6 | + |
| 7 | +```plaintext |
| 8 | +RuntimeError: CUDA error: no kernel image is available for execution on the device |
| 9 | +``` |
| 10 | + |
| 11 | +The failures are silent or cryptic, and none of them are documented in the existing tutorials. This guide walks through the exact image pin and flags needed to get single-GPU and multi-GPU (tensor-parallel) deployments running on V100 hardware, plus a reference table of the errors we hit and how we fixed them. |
| 12 | + |
| 13 | +All configurations here were validated on a bare-metal OpenShift (OKD 4.19, Kubernetes v1.32.8) cluster with `vllm-stack-0.1.11`, on Tesla V100 SXM2 16GB nodes, serving `Qwen2.5-0.5B/7B/14B-Instruct`. |
| 14 | + |
| 15 | +## Table of Contents |
| 16 | + |
| 17 | +- [Tutorial: Deploying on Tesla V100 and Other Legacy (Volta, CC 7.0) GPUs](#tutorial-deploying-on-tesla-v100-and-other-legacy-volta-cc-70-gpus) |
| 18 | + - [Introduction](#introduction) |
| 19 | + - [Table of Contents](#table-of-contents) |
| 20 | + - [Prerequisites](#prerequisites) |
| 21 | + - [Why V100 Needs Special Handling](#why-v100-needs-special-handling) |
| 22 | + - [Step 1: Single GPU Deployment (TP=1)](#step-1-single-gpu-deployment-tp1) |
| 23 | + - [Step 2: Multi-GPU Deployment (TP=2 and TP=4)](#step-2-multi-gpu-deployment-tp2-and-tp4) |
| 24 | + - [Step 3: Common Errors and Fixes](#step-3-common-errors-and-fixes) |
| 25 | + - [Step 4: Benchmark Reference Numbers](#step-4-benchmark-reference-numbers) |
| 26 | + - [Conclusion](#conclusion) |
| 27 | + |
| 28 | +## Prerequisites |
| 29 | + |
| 30 | +- A Kubernetes environment with GPU support, as set up in the [00-install-kubernetes-env tutorial](00-install-kubernetes-env.md). |
| 31 | +- One or more nodes with **Tesla V100 (or other Volta, CC 7.0) GPUs**, with the NVIDIA device plugin and `nvidia` runtime class installed. |
| 32 | +- Helm installed on your system. |
| 33 | +- Access to a HuggingFace token (`HF_TOKEN`) if your model is gated. The Qwen2.5 models used here are public. |
| 34 | +- Familiarity with [01-minimal-helm-installation.md](01-minimal-helm-installation.md) and [02-basic-vllm-config.md](02-basic-vllm-config.md). |
| 35 | + |
| 36 | +## Why V100 Needs Special Handling |
| 37 | + |
| 38 | +The Tesla V100 is a Volta GPU with **CUDA Compute Capability 7.0**. Modern vLLM builds and kernels increasingly target Ampere (CC 8.0) and newer, which leads to several hard constraints on Volta: |
| 39 | + |
| 40 | +- **No native BF16 datapath.** Volta has no hardware `bfloat16` support. If vLLM loads a model with the default `dtype=bfloat16`, kernel launches fail with `no kernel image is available for execution on the device`. You must run in FP16 by setting `dtype: "half"`. |
| 41 | +- **No FP8.** FP8 weights/KV cache (Hopper-era features) are unavailable on Volta. Stick to FP16. |
| 42 | +- **vLLM V1 engine assumes CC ≥ 8.0.** The newer V1 engine and its CUDA-graph / FlashAttention paths are not reliable on Volta. You must fall back to the legacy **V0 engine** (`v0: "1"`, which the chart maps to `VLLM_USE_V1=0`). |
| 43 | +- **Attention backend.** On Volta the supported attention path is XFormers (FlashAttention-2/3 require Ampere+). The V0 engine selects an appropriate Volta-compatible backend automatically once you are on the pinned image and FP16. |
| 44 | +- **The `latest` image dropped Volta.** As of vLLM **v0.9+**, the published `vllm/vllm-openai:latest` image no longer ships kernels compiled for CC < 8.0. You must **pin the image tag to `v0.8.5`**, the last release with Volta-compatible CUDA kernels. |
| 45 | + |
| 46 | +Putting it together, every V100 deployment in this tutorial pins `tag: "v0.8.5"`, sets `dtype: "half"`, forces `v0: "1"`, passes `--enforce-eager` (CUDA-graph capture is unstable on Volta with this image), and sets `VLLM_ENABLE_CUDA_COMPATIBILITY=1` to allow PTX JIT for forward compatibility. |
| 47 | + |
| 48 | +## Step 1: Single GPU Deployment (TP=1) |
| 49 | + |
| 50 | +Start with the smallest model to confirm the image and flags are correct before scaling out. The example file is [`tutorials/assets/values-25-v100-single-gpu.yaml`](assets/values-25-v100-single-gpu.yaml), serving `Qwen2.5-0.5B-Instruct` on a single V100. |
| 51 | + |
| 52 | +```yaml |
| 53 | +servingEngineSpec: |
| 54 | + runtimeClassName: "nvidia" |
| 55 | + strategy: |
| 56 | + type: Recreate |
| 57 | + modelSpec: |
| 58 | + - name: "qwen05b" |
| 59 | + repository: "vllm/vllm-openai" |
| 60 | + tag: "v0.8.5" |
| 61 | + modelURL: "Qwen/Qwen2.5-0.5B-Instruct" |
| 62 | + replicaCount: 1 |
| 63 | + requestCPU: 6 |
| 64 | + requestMemory: "16Gi" |
| 65 | + requestGPU: 1 |
| 66 | + pvcStorage: "20Gi" |
| 67 | + pvcAccessMode: |
| 68 | + - ReadWriteOnce |
| 69 | + vllmConfig: |
| 70 | + dtype: "half" |
| 71 | + v0: "1" |
| 72 | + tensorParallelSize: 1 |
| 73 | + maxModelLen: 16384 |
| 74 | + gpuMemoryUtilization: 0.9 |
| 75 | + extraArgs: |
| 76 | + - "--enforce-eager" |
| 77 | + env: |
| 78 | + - name: VLLM_ENABLE_CUDA_COMPATIBILITY |
| 79 | + value: "1" |
| 80 | + nodeSelectorTerms: |
| 81 | + - matchExpressions: |
| 82 | + - key: "kubernetes.io/hostname" |
| 83 | + operator: "In" |
| 84 | + values: |
| 85 | + - "gpu-node-01" |
| 86 | +``` |
| 87 | +
|
| 88 | +**Why each V100-specific field is needed:** |
| 89 | +
|
| 90 | +- **`tag: "v0.8.5"`** — the last image with kernels compiled for Compute Capability 7.0. `latest` (v0.9+) will not run on Volta. |
| 91 | +- **`vllmConfig.dtype: "half"`** — Volta has no BF16 hardware. Forcing FP16 avoids the `no kernel image available` crash you get from the default `bfloat16`. |
| 92 | +- **`vllmConfig.v0: "1"`** — selects the legacy V0 engine (the chart sets `VLLM_USE_V1=0`). The V1 engine assumes CC ≥ 8.0. |
| 93 | +- **`vllmConfig.tensorParallelSize: 1`** — a single 16GB V100 is enough for the 0.5B model. |
| 94 | +- **`extraArgs: ["--enforce-eager"]`** — disables CUDA-graph capture, which is unreliable on Volta with this image. The chart has no `enforceEager` field, so this must be passed through `extraArgs`. |
| 95 | +- **`env: VLLM_ENABLE_CUDA_COMPATIBILITY=1`** — enables PTX JIT so kernels can run on Volta via CUDA forward compatibility. |
| 96 | +- **`nodeSelectorTerms`** — pins the pod to a known V100 host via node affinity. Use `nodeSelectorTerms`, **not** `nodeSelector`; the chart only renders `nodeSelectorTerms`. |
| 97 | + |
| 98 | +Deploy it: |
| 99 | + |
| 100 | +```bash |
| 101 | +helm repo add vllm https://vllm-project.github.io/production-stack |
| 102 | +helm install vllm vllm/vllm-stack -f tutorials/assets/values-25-v100-single-gpu.yaml |
| 103 | +``` |
| 104 | + |
| 105 | +Verify the pod is running and check the logs to confirm the V0 engine and FP16 are active: |
| 106 | + |
| 107 | +```bash |
| 108 | +kubectl get pods |
| 109 | +kubectl logs -f <vllm-pod-name> | grep -Ei "dtype|engine|compute capability" |
| 110 | +``` |
| 111 | + |
| 112 | +Refer to Step 3 in [01-minimal-helm-installation.md](01-minimal-helm-installation.md) for querying the deployed service. |
| 113 | + |
| 114 | +## Step 2: Multi-GPU Deployment (TP=2 and TP=4) |
| 115 | + |
| 116 | +Models that do not fit in a single 16GB V100 must be sharded across GPUs with tensor parallelism. A 7B model in FP16 needs ~14GB just for weights (leaving no room for the KV cache), so it needs **TP=2** minimum; a 14B model needs **TP=4**. |
| 117 | + |
| 118 | +The example files are [`tutorials/assets/values-25-v100-multi-gpu-tp2.yaml`](assets/values-25-v100-multi-gpu-tp2.yaml) (`Qwen2.5-7B-Instruct`, TP=2) and [`tutorials/assets/values-25-v100-multi-gpu-tp4.yaml`](assets/values-25-v100-multi-gpu-tp4.yaml) (`Qwen2.5-14B-Instruct`, TP=4). |
| 119 | + |
| 120 | +TP=2 snippet: |
| 121 | + |
| 122 | +```yaml |
| 123 | +servingEngineSpec: |
| 124 | + runtimeClassName: "nvidia" |
| 125 | + strategy: |
| 126 | + type: Recreate |
| 127 | + modelSpec: |
| 128 | + - name: "qwen7b" |
| 129 | + repository: "vllm/vllm-openai" |
| 130 | + tag: "v0.8.5" |
| 131 | + modelURL: "Qwen/Qwen2.5-7B-Instruct" |
| 132 | + replicaCount: 1 |
| 133 | + requestCPU: 8 |
| 134 | + requestMemory: "32Gi" |
| 135 | + requestGPU: 2 |
| 136 | + pvcStorage: "50Gi" |
| 137 | + pvcAccessMode: |
| 138 | + - ReadWriteOnce |
| 139 | + vllmConfig: |
| 140 | + dtype: "half" |
| 141 | + v0: "1" |
| 142 | + tensorParallelSize: 2 |
| 143 | + maxModelLen: 8192 |
| 144 | + gpuMemoryUtilization: 0.9 |
| 145 | + extraArgs: |
| 146 | + - "--enforce-eager" |
| 147 | + env: |
| 148 | + - name: VLLM_ENABLE_CUDA_COMPATIBILITY |
| 149 | + value: "1" |
| 150 | + shmSize: "4Gi" |
| 151 | + nodeSelectorTerms: |
| 152 | + - matchExpressions: |
| 153 | + - key: "kubernetes.io/hostname" |
| 154 | + operator: "In" |
| 155 | + values: |
| 156 | + - "gpu-node-01" |
| 157 | +``` |
| 158 | + |
| 159 | +In addition to the single-GPU requirements, multi-GPU adds the following: |
| 160 | + |
| 161 | +- **`tensorParallelSize` must live in `vllmConfig`, NOT in `extraArgs`.** This is the easiest mistake to make. The chart renders `vllmConfig.tensorParallelSize` into the `--tensor-parallel-size` flag for you. If you instead add `--tensor-parallel-size` (or `--tensor_parallel_size`) to `extraArgs`, the value is **silently ignored**, tensor parallelism stays at 1, and the pod then OOMs while loading the model. Always set it in `vllmConfig`. |
| 162 | +- **`shmSize: "4Gi"`** — tensor-parallel all-reduce uses NCCL, which communicates between the per-GPU worker processes through `/dev/shm`. The Kubernetes default `/dev/shm` (64Mi) is too small for multi-GPU NCCL and causes init hangs or bus errors. The chart already mounts a 20Gi `tmpfs` whenever `tensorParallelSize` is set, so the default is sufficient; here we pin a smaller, resource-friendly `4Gi` that is still plenty for these models. |
| 163 | +- **`strategy.type: Recreate`** — set at the `servingEngineSpec` level. This is required on GPU nodes. With the default `RollingUpdate`, on a redeploy Kubernetes creates the new pod **before** terminating the old one. The new pod then waits for GPUs the old pod still holds, while the old pod is not torn down until the new one is `Ready` — a **deadlock** that leaves both pods `Pending` until you manually delete the old one. `Recreate` tears the old pod down first, releasing the GPUs. |
| 164 | + |
| 165 | +For **TP=4**, the same pattern applies with `requestGPU: 4` and `tensorParallelSize: 4` (see the TP=4 asset file). |
| 166 | + |
| 167 | +**NVLink topology awareness.** Tensor parallelism generates all-reduce traffic on every layer, so inter-GPU bandwidth matters. Place the parallel group on directly NVLink-connected GPUs: |
| 168 | + |
| 169 | +- On our cluster, `gpu-node-01` has 3x V100 in an NVLink "NV2" triangle, so any two GPUs on that host are directly NVLink-connected — a good fit for a **TP=2** pair. |
| 170 | +- `gpu-node-02` has 4x V100 in a **full NVLink mesh** (every GPU connected to every other), which is what **TP=4** wants. |
| 171 | + |
| 172 | +If the TP group spans GPUs that are only connected over PCIe, the all-reduce falls back to PCIe and throughput drops sharply. You can inspect the topology on a node with `nvidia-smi topo -m`. |
| 173 | + |
| 174 | +Deploy and verify the same way as Step 1: |
| 175 | + |
| 176 | +```bash |
| 177 | +helm install vllm vllm/vllm-stack -f tutorials/assets/values-25-v100-multi-gpu-tp2.yaml |
| 178 | +kubectl get pods |
| 179 | +``` |
| 180 | + |
| 181 | +## Step 3: Common Errors and Fixes |
| 182 | + |
| 183 | +These are the failures we hit on our test cluster and the exact fix for each. |
| 184 | + |
| 185 | +| Symptom / Error | Root Cause | Fix | |
| 186 | +| --- | --- | --- | |
| 187 | +| `RuntimeError: CUDA error: no kernel image is available for execution on the device` | Model loaded as `bfloat16` (the default); Volta has no BF16 kernels. | Set `vllmConfig.dtype: "half"`. | |
| 188 | +| `CUDA error: invalid device function` | Using `vllm/vllm-openai:latest` (v0.9+), which dropped CC < 8.0 kernels. | Pin `tag: "v0.8.5"`. | |
| 189 | +| Pod OOM / killed while loading a 7B+ model on a single GPU | The model does not fit in 16GB of V100 memory at TP=1. | Use TP=2 (7B) or TP=4 (14B) via `requestGPU` + `tensorParallelSize`. | |
| 190 | +| Pod scheduled on the wrong (non-V100) node | Used `nodeSelector`, which this chart does not render. | Use `nodeSelectorTerms` (node affinity) instead. | |
| 191 | +| Both old and new pods stuck `Pending` after a redeploy | Default `RollingUpdate` starts the new pod before the old one frees its GPUs — GPU deadlock. | Set `servingEngineSpec.strategy.type: Recreate`. | |
| 192 | +| Engine reports `tensor_parallel_size=1` even though you set it | `--tensor-parallel-size` was placed in `extraArgs`, where it is silently ignored. | Set `tensorParallelSize` inside `vllmConfig`. | |
| 193 | +| NCCL init hang or bus error on multi-GPU startup | `/dev/shm` too small (Kubernetes default 64Mi). | Set `shmSize` (e.g. `4Gi`); the chart's `20Gi` default also works. | |
| 194 | + |
| 195 | +## Step 4: Benchmark Reference Numbers |
| 196 | + |
| 197 | +The following throughput numbers are from real runs on our V100 SXM2 16GB nodes with the configurations in this tutorial. Use them as a sanity check that your deployment is in the right ballpark, not as guaranteed performance. |
| 198 | + |
| 199 | +| Model | TP | Approx. throughput | Notes | |
| 200 | +| --- | --- | --- | --- | |
| 201 | +| `Qwen2.5-0.5B-Instruct` | 1 | ~435 tok/s/GPU | Single V100, fits easily. | |
| 202 | +| `Qwen2.5-7B-Instruct` | 2 | ~155 tok/s/GPU | NVLink pair on `gpu-node-01`. | |
| 203 | +| `Qwen2.5-14B-Instruct` | 4 | ~56 tok/s/GPU | Full NVLink mesh on `gpu-node-02`. | |
| 204 | + |
| 205 | +Per-GPU throughput drops as the tensor-parallel degree grows because all-reduce communication overhead increases with the size of the TP group — this is expected, and is exactly why NVLink topology matters. |
| 206 | + |
| 207 | +For interactive serving, regardless of model size, plan for roughly **8–16 concurrent requests per deployment** before latency-sensitive (interactive SLO) requests start to degrade. Scale out with additional replicas (and additional GPUs) rather than pushing a single deployment past that range. |
| 208 | + |
| 209 | +## Conclusion |
| 210 | + |
| 211 | +Volta-generation GPUs like the Tesla V100 are still capable inference accelerators, but they require deliberate configuration on the modern vLLM Production Stack: pin the image to `v0.8.5`, run in FP16 (`dtype: "half"`), force the V0 engine (`v0: "1"`), use `--enforce-eager`, and set `VLLM_ENABLE_CUDA_COMPATIBILITY=1`. For multi-GPU, keep `tensorParallelSize` in `vllmConfig`, set an adequate `shmSize` for NCCL, use the `Recreate` strategy to avoid GPU deadlocks, and align the tensor-parallel group with the NVLink topology. With these settings, single-GPU and tensor-parallel deployments run reliably on CC 7.0 hardware. |
0 commit comments