Skip to content

Commit 4ff2a80

Browse files
authored
Update vLLM documentation with test environment and install
Added test environment details and installation instructions for vLLM.
1 parent de2752f commit 4ff2a80

1 file changed

Lines changed: 7 additions & 0 deletions

File tree

docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,13 @@ We tested `vLLM` and `llama-cpp` (the inference framework behind `ollama`) on To
1111
|`vLLM`|~4689.6|~48.0|Best for Batch/Research|
1212
|`llama-cpp`|~115.0|~280.0|Best for Single User|
1313

14+
### Test Environment
15+
GPU: NVIDIA L40S
16+
17+
`vLLM`: 0.13.0
18+
19+
`Ollama` (llama-cpp backend): 0.14.2
20+
1421
## vLLM Installation Instructions
1522
Create a `vLLM` directory in your /scratch directory, then install the vLLM image:
1623
```

0 commit comments

Comments
 (0)