Skip to content

Commit 49822fd

Browse files
hmhv1222hmhv1222
committed
Update model lists for TRTLLM inference on A4 & A4X in main ReadMe and individual A4/A4X ReadMes
1 parent dc0a187 commit 49822fd

3 files changed

Lines changed: 42 additions & 11 deletions

File tree

‎README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,28 @@ Models | GPU Machine Type
8282
| ---------------- | ---------------- | --------- | ------------------- | ------------ | ------------------ |
8383
| **DeepSeek R1 671B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | vLLM | Inference | GKE | [Link](./inference/a4/single-host-serving/vllm/README.md)
8484
| **DeepSeek R1 671B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | SGLang | Inference | GKE | [Link](./inference/a4/single-host-serving/sglang/README.md)
85+
| **DeepSeek R1 671B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
86+
| **Llama 3.1 405B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
87+
| **Qwen 2.5 VL 7B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
88+
| **Qwen 3 235B A22B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
89+
| **Qwen 3 32B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
90+
91+
### Inference benchmarks A4X
92+
93+
| Models | GPU Machine Type | Framework | Workload Type | Orchestrator | Link to the recipe |
94+
| ---------------- | ---------------- | --------- | ------------------- | ------------ | ------------------ |
95+
| **DeepSeek R1 671B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | vLLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/vllm/README.md)
96+
| **Wan2.2 T2V A14B Diffusers** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | SGLang | Inference | GKE | [Link](./inference/a4x/single-host-serving/sglang/README.md)
97+
| **Wan2.2 I2V A14B Diffusers** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | SGLang | Inference | GKE | [Link](./inference/a4x/single-host-serving/sglang/README.md)
98+
| **DeepSeek R1 671B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md) <br> <br> [Link for Using Google Cloud Storage (GCS) as Storage Option]((./inference/a4x/single-host-serving/tensorrt-llm-gcs/README.md)) <br> <br> [Link for Using Lustre as Storage Option]((./inference/a4x/single-host-serving/tensorrt-llm-lustre/README.md))
99+
| **Llama 3.1 405B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
100+
| **Llama 3.1 70B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
101+
| **Llama 3.1 8B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
102+
| **Qwen 2.5 VL 7B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
103+
| **Qwen 3 235B A22B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
104+
| **Qwen 3 32B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
105+
| **Qwen 3 4B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
106+
85107

86108
### Inference benchmarks G4
87109

‎inference/a4/single-host-serving/tensorrt-llm/README.md‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -205,12 +205,17 @@ Example model configuration YAML files included in this repo only show a certain
205205

206206
| Model Name | Hugging Face ID | Configuration File | Release Name Suffix |
207207
| :--- | :--- | :--- | :--- |
208-
| **DeepSeek-R1 671B** | `nvidia/DeepSeek-R1-NVFP4-v2` | `deepseek-r1-nvfp4.yaml` | `deepseek-r1` |
209-
| **Qwen 3 235B A22B FP4** | `nvidia/Qwen3-235B-A22B-NVFP4` | `qwen3-235b-a22b-nvfp4.yaml` | `qwen3-235b-a22b` |
208+
| **DeepSeek R1 671B** | `nvidia/DeepSeek-R1-NVFP4-v2` | `deepseek-r1-nvfp4.yaml` | `deepseek-r1` |
209+
| **Llama 3.1 405B (FP8)** | `meta-llama/Llama-3.1-405B-Instruct-FP8` | `llama-3.1-405b.yaml` | `llama-3-1-405b` |
210+
| **Llama 3.1 405B (NVFP4)** | `nvidia/Llama-3.1-405B-Instruct-NVFP4` | `llama-3.1-405b.yaml` | `llama-3-1-405b` |
211+
| **Qwen 2.5 VL 7B (FP8)** | `Qwen/Qwen2.5-VL-7B-Instruct` | `qwen2-5-vl-7b-fp8.yaml` | `qwen2-5-vl-7b` |
212+
| **Qwen 2.5 VL 7B (NVFP4)** | `nvidia/Qwen2.5-VL-7B-Instruct-NVFP4` | `qwen2-5-vl-7b-nvfp4.yaml` | `qwen2-5-vl-7b` |
213+
| **Qwen 3 235B A22B (FP8)** | `Qwen/Qwen3-235B-A22B-FP8` | `qwen3-235b-a22b-fp8.yaml` | `qwen3-235b-a22b` |
214+
| **Qwen 3 235B A22B (NVFP4)** | `nvidia/Qwen3-235B-A22B-NVFP4` | `qwen3-235b-a22b-nvfp4.yaml` | `qwen3-235b-a22b` |
210215
| **Qwen 3 32B** | `Qwen/Qwen3-32B` | `qwen3-32b.yaml` | `qwen3-32b` |
211216

212217
> [!TIP]
213-
> **DeepSeek-R1 671B** uses Nvidia's pre-quantized FP4 checkpoint. For more information, see the [Hugging Face model card](https://huggingface.co/nvidia/DeepSeek-R1-NVFP4-v2).
218+
> **DeepSeek R1 671B** uses Nvidia's pre-quantized FP4 checkpoint. For more information, see the [Hugging Face model card](https://huggingface.co/nvidia/DeepSeek-R1-NVFP4-v2).
214219
215220
> [!TIP]
216221
> You can use the [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq) to quantize these models to FP8 or NVFP4 for improved performance.
@@ -225,7 +230,7 @@ The recipe uses [`trtllm-bench`](https://github.com/NVIDIA/TensorRT-LLM/blob/mai
225230
1. **Configure model-specific variables.** Choose a model from the [table above](#supported-models) and set the variables:
226231

227232
```bash
228-
# Example for DeepSeek-R1 NVFP4
233+
# Example for DeepSeek R1 NVFP4
229234
export HF_MODEL_ID="nvidia/DeepSeek-R1-NVFP4-v2"
230235
export CONFIG_FILE="deepseek-r1-nvfp4.yaml"
231236
export RELEASE_NAME="$USER-serving-deepseek-r1"
@@ -269,7 +274,7 @@ After the model is deployed via Helm as described in the sections [above](#run-t
269274
Check the status of your deployment. Replace the name if you deployed a different model.
270275

271276
```bash
272-
# Example for DeepSeek-R1 671B
277+
# Example for DeepSeek R1 671B
273278
kubectl get deployment/$USER-serving-deepseek-r1
274279
```
275280

‎inference/a4x/single-host-serving/tensorrt-llm/README.md‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -205,16 +205,20 @@ Example model configuration YAML files included in this repo only show a certain
205205

206206
| Model Name | Hugging Face ID | Configuration File | Release Name Suffix |
207207
| :--- | :--- | :--- | :--- |
208-
| **DeepSeek-R1 671B** | `nvidia/DeepSeek-R1-NVFP4-v2` | `deepseek-r1-nvfp4.yaml` | `deepseek-r1` |
209-
| **Llama 3.1 405B NVFP4** | `nvidia/Llama-3.1-405B-Instruct-NVFP4` | `llama-3.1-405b.yaml` | `llama-3-1-405b-nvfp4` |
210-
| **Llama 3.1 405B FP8** | `meta-llama/Llama-3.1-405B-Instruct-FP8` | `llama-3.1-405b.yaml` | `llama-3-1-405b-fp8` |
208+
| **DeepSeek R1 671B** | `nvidia/DeepSeek-R1-NVFP4-v2` | `deepseek-r1-nvfp4.yaml` | `deepseek-r1` |
209+
| **Llama 3.1 405B (FP8)** | `meta-llama/Llama-3.1-405B-Instruct-FP8` | `llama-3.1-405b.yaml` | `llama-3-1-405b` |
210+
| **Llama 3.1 405B (NVFP4)** | `nvidia/Llama-3.1-405B-Instruct-NVFP4` | `llama-3.1-405b.yaml` | `llama-3-1-405b` |
211211
| **Llama 3.1 70B** | `meta-llama/Llama-3.1-70B-Instruct` | `llama-3.1-70b.yaml` | `llama-3-1-70b` |
212212
| **Llama 3.1 8B** | `meta-llama/Llama-3.1-8B-Instruct` | `llama-3.1-8b.yaml` | `llama-3-1-8b` |
213+
| **Qwen 2.5 VL 7B (FP8)** | `Qwen/Qwen2.5-VL-7B-Instruct` | `qwen2-5-vl-7b-fp8.yaml` | `qwen2-5-vl-7b` |
214+
| **Qwen 2.5 VL 7B (NVFP4)** | `nvidia/Qwen2.5-VL-7B-Instruct-NVFP4` | `qwen2-5-vl-7b-nvfp4.yaml` | `qwen2-5-vl-7b` |
215+
| **Qwen 3 235B A22B (FP8)** | `Qwen/Qwen3-235B-A22B-FP8` | `qwen3-235b-a22b-fp8.yaml` | `qwen3-235b-a22b` |
216+
| **Qwen 3 235B A22B (NVFP4)** | `nvidia/Qwen3-235B-A22B-NVFP4` | `qwen3-235b-a22b-nvfp4.yaml` | `qwen3-235b-a22b` |
213217
| **Qwen 3 32B** | `Qwen/Qwen3-32B` | `qwen3-32b.yaml` | `qwen3-32b` |
214218
| **Qwen 3 4B** | `Qwen/Qwen3-4B` | `qwen3-4b.yaml` | `qwen3-4b` |
215219

216220
> [!TIP]
217-
> **DeepSeek-R1 671B** uses Nvidia's pre-quantized FP4 checkpoint. For more information, see the [Hugging Face model card](https://huggingface.co/nvidia/DeepSeek-R1-NVFP4-v2).
221+
> **DeepSeek R1 671B** uses Nvidia's pre-quantized FP4 checkpoint. For more information, see the [Hugging Face model card](https://huggingface.co/nvidia/DeepSeek-R1-NVFP4-v2).
218222
219223
> [!TIP]
220224
> You can use the [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq) to quantize these models to FP8 or NVFP4 for improved performance.
@@ -229,7 +233,7 @@ The recipe uses [`trtllm-bench`](https://github.com/NVIDIA/TensorRT-LLM/blob/mai
229233
1. **Configure model-specific variables.** Choose a model from the [table above](#supported-models) and set the variables:
230234

231235
```bash
232-
# Example for DeepSeek-R1 NVFP4
236+
# Example for DeepSeek R1 NVFP4
233237
export HF_MODEL_ID="nvidia/DeepSeek-R1-NVFP4-v2"
234238
export CONFIG_FILE="deepseek-r1-nvfp4.yaml"
235239
export RELEASE_NAME="$USER-serving-deepseek-r1"
@@ -273,7 +277,7 @@ After the model is deployed via Helm as described in the sections [above](#run-t
273277
Check the status of your deployment. Replace the name if you deployed a different model.
274278

275279
```bash
276-
# Example for DeepSeek-R1 671B
280+
# Example for DeepSeek R1 671B
277281
kubectl get deployment/$USER-serving-deepseek-r1
278282
```
279283

0 commit comments

Comments
 (0)