Skip to content

Commit 65ccbf4

Browse files
committed
Add benchmarking configs warning paragraphs in TRTLLM inference ReadMe
1 parent e6445b7 commit 65ccbf4

2 files changed

Lines changed: 8 additions & 0 deletions

File tree

  • inference

inference/a4/single-host-serving/tensorrt-llm/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,10 @@ kubectl create secret generic hf-secret \
199199

200200
This recipe supports the following models. You can easily swap between them by changing the environment variables in the next step.
201201

202+
Running TRTLLM inference benchmarking on these models are only tested and validated on A4 GKE nodes with certain combination of TP, PP, EP, number of GPU chips, input & output sequence length, precision, etc.
203+
204+
Example model configuration YAML files included in this repo only show a certain combination of parallelism hyperparameters and configs for benchmarking purposes. Input and output length in `gpu-recipes/inference/a4/single-host-serving/tensorrt-llm/values.yaml` need to be adjusted according to the model and its configs.
205+
202206
| Model Name | Hugging Face ID | Configuration File | Release Name Suffix |
203207
| :--- | :--- | :--- | :--- |
204208
| **DeepSeek-R1 671B** | `nvidia/DeepSeek-R1-NVFP4-v2` | `deepseek-r1-nvfp4.yaml` | `deepseek-r1` |

inference/a4x/single-host-serving/tensorrt-llm/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,10 @@ kubectl create secret generic hf-secret \
199199

200200
This recipe supports the following models. You can easily swap between them by changing the environment variables in the next step.
201201

202+
Running TRTLLM inference benchmarking on these models are only tested and validated on A4X GKE nodes with certain combination of TP, PP, EP, number of GPU chips, input & output sequence length, precision, etc.
203+
204+
Example model configuration YAML files included in this repo only show a certain combination of parallelism hyperparameters and configs for benchmarking purposes. Input and output length in `gpu-recipes/inference/a4x/single-host-serving/tensorrt-llm/values.yaml` need to be adjusted according to the model and its configs.
205+
202206
| Model Name | Hugging Face ID | Configuration File | Release Name Suffix |
203207
| :--- | :--- | :--- | :--- |
204208
| **DeepSeek-R1 671B** | `nvidia/DeepSeek-R1-NVFP4-v2` | `deepseek-r1-nvfp4.yaml` | `deepseek-r1` |

0 commit comments

Comments
 (0)