docs: update & add new AWS DLCs#2571
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
pagezyhf
left a comment
pagezyhf
left a comment
There was a problem hiding this comment.
LGTM!
I think we should also update the FAQ:
How to choose the right inference container for my use case?can be removed?How to find the URI of my container?I'm not sure it's still what we should recommend.How to find the URI of my container but simpler?we should migrate to v3.(https://sagemaker.readthedocs.io/en/stable/inference/index.html#migration-from-v2)
|
Hey @pagezyhf! Yes, maybe I can update the |
| from sagemaker.core import image_uris | ||
|
|
||
| The `image_uris.retrieve` helper from `sagemaker.core` is not always up to date but it is much simpler than reconstructing the image URI yourself. The processor (CPU/GPU/Neuron) is inferred from the `instance_type`. | ||
| AVAILABLE_FRAMEWORKS = [ |
There was a problem hiding this comment.
are you sure it works for all of those families? last time I checked it didn't
There was a problem hiding this comment.
Yes, I checked. This is the output if you go through them:
Image for huggingface-tei: 683313688378.dkr.ecr.us-east-1.amazonaws.com/tei:2.0.1-tei1.8.2-gpu-py310-cu122-ubuntu22.04
Image for huggingface-llamacpp: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-llamacpp:b9522-gpu-cu130-ubuntu24.04
Image for huggingface-vllm: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04
Image for huggingface-vllm-omni: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-vllm-omni:0.20.0-transformers5.8.1-gpu-py312-cu130-amzn2023
Image for huggingface-sglang: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-sglang:0.5.8-transformers4.57.3-gpu-py312-cu129-ubuntu24.04
@dwarez updated them all on aws/sagemaker-python-sdk#5960
There was a problem hiding this comment.
I'm just seeing that TEI is not updated in the SDK to 1.9.3, I'll create a PR for that too 👍
There was a problem hiding this comment.
we'll need to make sure those are up to date in the sagemakersdk too, so one more thing to maintain but 👍
|
|
||
| > [!NOTE] | ||
| > These docs and examples use the [SageMaker Python SDK v3](https://github.com/aws/sagemaker-python-sdk), which introduces a new framework-agnostic API built around `ModelBuilder` (inference) and `ModelTrainer` (training), replacing the v2 `HuggingFaceModel` and `HuggingFace` classes. Install it with `pip install "sagemaker>=3.0.0"`. In v2 these URIs were retrieved with `get_huggingface_llm_image_uri`, which has been removed. | ||
| If you just want to use the default container for a given model, you can also rely on the SageMaker SDK `ModelBuilder`, which will automatically choose the correct container for you: |
There was a problem hiding this comment.
@dwarez curious about your thoughts here as you spent time looking at this logic.
There was a problem hiding this comment.
that's correct, model builder uses the routing logic we defined in the sdk. Then it's up to us to specify the logic and keep it updated as we publish more dlcs
4 participants
Description
This PR adds docs for 2 new AWS DLCs:
Additionally, it updates the vLLM and SGLang versions to the latest available container. It also updates the FAQ about how to obtain the image URI for your model using the SageMaker Python SDK.
Note
Low Risk
Documentation-only changes to container URIs and examples; no runtime or application code is modified.
Overview
Updates the SageMaker available DLCs page with newer inference images and two new serving stacks.
Inference tables: bumps the documented vLLM GPU image from 0.17.0 to 0.21.0 (transformers 5.8.1, cu130) and SGLang from 0.5.8 to 0.5.12; refreshes TEI CPU/GPU URIs to tei 1.9.3 on Ubuntu 24.04. Adds sections for vLLM Omni (multimodal, 0.20.0) and Llama.cpp (CPU and GPU, tag b9522).
FAQ: replaces manual ECR account/region assembly and the old multi-
printimage_uris.retrieveexamples with a singleimage_uris.retrievepattern listing frameworks including vLLM Omni and Llama.cpp, plus an optionalModelBuildersnippet for automatic container selection, with a note to verify URIs against this doc when the SDK lags.Reviewed by Cursor Bugbot for commit 9e7f3f2. Bugbot is set up for automated code reviews on this repo. Configure here.