@@ -29,14 +29,24 @@ For inference, there is a general-purpose PyTorch inference DLC, for serving mod
2929| 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:2.6.0-transformers4.51.3-gpu-py312-cu124-ubuntu22.04 | GPU |
3030| 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference-neuronx:2.8.0-transformers4.55.4-neuronx-py310-sdk2.26.0-ubuntu22.04 | Neuron |
3131
32- ### LLM DLC
3332
34- For high-performance serving of text generation models, there is the LLM DLC, available on GPU and AWS AI chips.
33+ ### Hugging Face vLLM Inference DLC
34+
35+ In case you want to serve text generation models with vLLM, there are specific DLCs available for GPU and AWS AI chips.
36+
37+ | vLLM version | Container URI | Accelerator |
38+ | -------------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------- |
39+ | 0.14.0 | 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-vllm:0.14-gpu-py312-cu129-ubuntu22.04-v1 | GPU |
40+ | 0.11.0 | 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-vllm-inference-neuronx:0.11.0-optimum0.4.5-neuronx-py310-sdk2.26.1-ubuntu22.04 | Neuron |
41+
42+ ### Hugging Face SGLang Inference DLC
43+
44+ There is also a specific DLC for serving models with SGLang on GPU.
3545
3646| Container URI | Accelerator |
3747| -------------------------------------------------------------------------------------------------------------------------------- | ----------- |
38- | 763104351884.dkr.ecr.us-east-1 .amazonaws.com/huggingface-pytorch-tgi-inference:2.7.0-tgi3.3.6 -gpu-py311-cu124-ubuntu22 .04 | GPU |
39- | 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-vllm-inference-neuronx:0.11.0-optimum0.4.4-neuronx-py310-sdk2.26.1-ubuntu22.04 | Neuron |
48+ | 763104351884.dkr.ecr.us-west-2 .amazonaws.com/huggingface-sglang:0.5.8-transformers4.57.3 -gpu-py312-cu129-ubuntu24 .04 | GPU |
49+
4050
4151### Text Embedding Inference
4252
@@ -80,4 +90,4 @@ print(f"TEI CPU: {get_huggingface_llm_image_uri('huggingface-tei-cpu')}")
8090print (f " TGI Neuron: { get_huggingface_llm_image_uri(' huggingface-neuronx' )} " )
8191```
8292
83- For Pytorch Training and Pytorch Inference DLCs, there is no such utility.
93+ For Pytorch Training and Pytorch Inference DLCs, there is no such utility.
0 commit comments