Skip to content

Commit 9611817

Browse files
authored
Update vLLM documentation with additional model details
Expanded performance comparison of vLLM and llama-cpp for different models and workloads.
1 parent 0eaa66f commit 9611817

1 file changed

Lines changed: 6 additions & 6 deletions

File tree

docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,18 @@
44
[`vLLM`](https://docs.vllm.ai/en/latest/) is an easy-to-use library for LLM inference and serving which support a wide variety of models with optimized kernels ensuring optimal utilization of GPUs.
55

66
## Why `vLLM`?
7-
We tested `vLLM` and `llama-cpp` on Torch, and found `vLLM` performs better on Torch for `Qwen2.5-0.5B-Instruct` with `256` input and `256` output tokens.
7+
We tested `vLLM` and `llama-cpp` on Torch, and found `vLLM` performs better on Torch for `Qwen2.5-0.5B-Instruct` and `Qwen2.5-7B-Instruct` with `256` input and `256` output tokens. In terms of practical usage, vLLM is better suited for high-concurrency and batch workloads, while llama-cpp is more appropriate for lightweight or single-user scenarios.
88

9-
|Inference Server|Total Throughput (tok/s)|Median Latency (ms)|Recommendation
9+
|Model|Inference Server|Generated Throughput (tok/s)|Median Latency (ms)|
1010
|-----|-----|-----|-----|
11-
|`vLLM`|~2273|~890|Best for Batch/Research|
12-
|`llama-cpp`|~1312|~1440 |Best for Single User|
11+
|0.5B|`vLLM`|~2273|~890|
12+
|0.5B|`llama-cpp`|~1312|~1440 |
13+
|7B|`vLLM`|~354|~5780 |
14+
|7B|`llama-cpp`|~277|~7200 |
1315

1416
### Test Environment
1517
GPU: NVIDIA L40S
1618

17-
Model: `Qwen2.5-0.5B-Instruct`
18-
1919
Precision: FP16
2020

2121
Workload: 256 input / 256 output tokens

0 commit comments

Comments
 (0)