Skip to content

Commit a30266f

Browse files
authored
Enhance vLLM section with performance details
Expanded the comparison between vLLM and llama-cpp, highlighting performance advantages and integration with the Hugging Face/Torch ecosystem.
1 parent 9611817 commit a30266f

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
[`vLLM`](https://docs.vllm.ai/en/latest/) is an easy-to-use library for LLM inference and serving which support a wide variety of models with optimized kernels ensuring optimal utilization of GPUs.
55

66
## Why `vLLM`?
7-
We tested `vLLM` and `llama-cpp` on Torch, and found `vLLM` performs better on Torch for `Qwen2.5-0.5B-Instruct` and `Qwen2.5-7B-Instruct` with `256` input and `256` output tokens. In terms of practical usage, vLLM is better suited for high-concurrency and batch workloads, while llama-cpp is more appropriate for lightweight or single-user scenarios.
7+
We tested `vLLM` and `llama-cpp` on Torch, and found `vLLM` performs better on Torch for `Qwen2.5-0.5B-Instruct` and `Qwen2.5-7B-Instruct` with `256` input and `256` output tokens. Based on our results, vLLM retains a performance advantage over llama-cpp across both model sizes, although the relative gap decreases as the model size increases. In addition, vLLM integrates more naturally with the Hugging Face/Torch ecosystem in HPC environment, whereas llama-cpp requires more manual setup and compatibility adjustment.
88

99
|Model|Inference Server|Generated Throughput (tok/s)|Median Latency (ms)|
1010
|-----|-----|-----|-----|

0 commit comments

Comments
 (0)