Skip to content

Commit de2752f

Browse files
committed
minor polish
1 parent 48754ac commit de2752f

1 file changed

Lines changed: 1 addition & 7 deletions

File tree

docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,7 @@
44
[`vLLM`](https://docs.vllm.ai/en/latest/) is an easy-to-use library for LLM inference and serving which support a wide variety of models with optimized kernels ensuring optimal utilization of GPUs.
55

66
## Why `vLLM`?
7-
We tested `vLLM` and `llama-cpp` (the inference framework behind `ollama`) on Torch, and found `vLLM` performs better on Torch with the following model and token configuration:
8-
9-
Model: `Qwen2.5-7B-Instruct`
10-
11-
Prompt Tokens: `512`
12-
13-
Output Tokens: `256`
7+
We tested `vLLM` and `llama-cpp` (the inference framework behind `ollama`) on Torch, and found `vLLM` performs better on Torch for `Qwen2.5-7B-Instruct` with `512` input and `256` output tokens.
148

159
|Inference Server|Peak Throughput|Median Latency(ms)|Recommendation
1610
|-----|-----|-----|-----|

0 commit comments

Comments
 (0)