Skip to content

Commit c7c73e4

Browse files
ehcalabrespagezyhf
andauthored
Add AWS SM vLLM & SGLang Inference DLCs (#2277)
* docs: add AWS SM HF vLLM Inference DLC details * docs: update available DLCs for vLLM and SGLang * Update docs/sagemaker/source/dlcs/available.md Co-authored-by: pagezyhf <165770107+pagezyhf@users.noreply.github.com> --------- Co-authored-by: pagezyhf <165770107+pagezyhf@users.noreply.github.com>
1 parent 815580d commit c7c73e4

1 file changed

Lines changed: 15 additions & 5 deletions

File tree

docs/sagemaker/source/dlcs/available.md

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,14 +29,24 @@ For inference, there is a general-purpose PyTorch inference DLC, for serving mod
2929
| 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:2.6.0-transformers4.51.3-gpu-py312-cu124-ubuntu22.04 | GPU |
3030
| 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference-neuronx:2.8.0-transformers4.55.4-neuronx-py310-sdk2.26.0-ubuntu22.04 | Neuron |
3131

32-
### LLM DLC
3332

34-
For high-performance serving of text generation models, there is the LLM DLC, available on GPU and AWS AI chips.
33+
### Hugging Face vLLM Inference DLC
34+
35+
In case you want to serve text generation models with vLLM, there are specific DLCs available for GPU and AWS AI chips.
36+
37+
| vLLM version | Container URI | Accelerator |
38+
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------- |
39+
| 0.14.0 | 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-vllm:0.14-gpu-py312-cu129-ubuntu22.04-v1 | GPU |
40+
| 0.11.0 | 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-vllm-inference-neuronx:0.11.0-optimum0.4.5-neuronx-py310-sdk2.26.1-ubuntu22.04 | Neuron |
41+
42+
### Hugging Face SGLang Inference DLC
43+
44+
There is also a specific DLC for serving models with SGLang on GPU.
3545

3646
| Container URI | Accelerator |
3747
| -------------------------------------------------------------------------------------------------------------------------------- | ----------- |
38-
| 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.7.0-tgi3.3.6-gpu-py311-cu124-ubuntu22.04 | GPU |
39-
| 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-vllm-inference-neuronx:0.11.0-optimum0.4.4-neuronx-py310-sdk2.26.1-ubuntu22.04 | Neuron |
48+
| 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-sglang:0.5.8-transformers4.57.3-gpu-py312-cu129-ubuntu24.04 | GPU |
49+
4050

4151
### Text Embedding Inference
4252

@@ -80,4 +90,4 @@ print(f"TEI CPU: {get_huggingface_llm_image_uri('huggingface-tei-cpu')}")
8090
print(f"TGI Neuron: {get_huggingface_llm_image_uri('huggingface-neuronx')}")
8191
```
8292

83-
For Pytorch Training and Pytorch Inference DLCs, there is no such utility.
93+
For Pytorch Training and Pytorch Inference DLCs, there is no such utility.

0 commit comments

Comments
 (0)