Skip to content

Commit 25c10ba

Browse files
hmhv1222hmhv1222
authored
Merge pull request #146 from AI-Hypercomputer/mhvictorhau-20260303-a4-a4x-trtllm-benchmarking
A4 & A4X TRTLLM GKE single-host inference benchmarking recipes for DeepSeek R1, Llama 3.1 405B, Qwen 2.5 VL 7B, Qwen 3 235B A22B, Qwen 3 32B
2 parents efe118b + ab3c117 commit 25c10ba

19 files changed

Lines changed: 757 additions & 50 deletions

File tree

‎README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,28 @@ Models | GPU Machine Type
8282
| ---------------- | ---------------- | --------- | ------------------- | ------------ | ------------------ |
8383
| **DeepSeek R1 671B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | vLLM | Inference | GKE | [Link](./inference/a4/single-host-serving/vllm/README.md)
8484
| **DeepSeek R1 671B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | SGLang | Inference | GKE | [Link](./inference/a4/single-host-serving/sglang/README.md)
85+
| **DeepSeek R1 671B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
86+
| **Llama 3.1 405B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
87+
| **Qwen 2.5 VL 7B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
88+
| **Qwen 3 235B A22B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
89+
| **Qwen 3 32B** | [A4 (NVIDIA B200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4/single-host-serving/tensorrt-llm/README.md)
90+
91+
### Inference benchmarks A4X
92+
93+
| Models | GPU Machine Type | Framework | Workload Type | Orchestrator | Link to the recipe |
94+
| ---------------- | ---------------- | --------- | ------------------- | ------------ | ------------------ |
95+
| **DeepSeek R1 671B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | vLLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/vllm/README.md)
96+
| **Wan2.2 T2V A14B Diffusers** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | SGLang | Inference | GKE | [Link](./inference/a4x/single-host-serving/sglang/README.md)
97+
| **Wan2.2 I2V A14B Diffusers** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | SGLang | Inference | GKE | [Link](./inference/a4x/single-host-serving/sglang/README.md)
98+
| **DeepSeek R1 671B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md) <br> <br> [Link for Using Google Cloud Storage (GCS) as Storage Option]((./inference/a4x/single-host-serving/tensorrt-llm-gcs/README.md)) <br> <br> [Link for Using Lustre as Storage Option]((./inference/a4x/single-host-serving/tensorrt-llm-lustre/README.md))
99+
| **Llama 3.1 405B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
100+
| **Llama 3.1 70B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
101+
| **Llama 3.1 8B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
102+
| **Qwen 2.5 VL 7B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
103+
| **Qwen 3 235B A22B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
104+
| **Qwen 3 32B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
105+
| **Qwen 3 4B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md)
106+
85107

86108
### Inference benchmarks G4
87109

0 commit comments

Comments
 (0)