|
1 | 1 | # Introduction |
2 | 2 |
|
3 | | -Hugging Face built Deep Learning Containers (DLCs) for Amazon Web Services customers to run any of their machine learning workload in an optimized environment, with no configuration or maintenance on their part. These are Docker images pre-installed with deep learning frameworks and libraries such as 🤗 Transformers, 🤗 Datasets, and 🤗 Tokenizers. The DLCs allow you to directly serve and train any models, skipping the complicated process of building and optimizing your serving and training environments from scratch. |
| 3 | +Hugging Face, together with Amazon Web Services, builds and maintains Deep Learning Containers (DLCs) so you can run your machine learning workloads in an optimized environment with no configuration or maintenance on your part. These are Docker images pre-installed with popular frameworks and libraries such as 🤗 Transformers, 🤗 Datasets, and 🤗 Tokenizers, alongside high-performance serving engines. The DLCs let you serve and train models directly, skipping the complex process of building and optimizing your own environments from scratch. |
4 | 4 |
|
5 | | -The containers are publicly maintained, updated and released periodically by Hugging Face and the AWS team and available for all AWS customers within the AWS’s Elastic Container Registry. They can be used from any AWS service such as: |
6 | | -* **Amazon Sagemaker AI**: Amazon SageMaker AI is a fully managed machine learning (ML) platform for data scientists and developers to quickly and confidently build, train, and deploy ML models into a production-ready hosted environment. |
7 | | -* **Amazon Bedrock**: Amazon Bedrock is a fully managed service that makes high-performing foundation models (FMs) from leading AI companies and Amazon available for your use through a unified API to build generative AI applications. |
8 | | -* **Amazon Elastic Kubernetes Service (EKS)**: Amazon EKS is the premiere platform for running Kubernetes clusters in the AWS cloud. |
9 | | -* **Amazon Elastic Container Service (ECS)**: Amazon ECS is a fully managed container orchestration service that helps you easily deploy, manage, and scale containerized applications. |
10 | | -* **Amazon Elastic Compute Cloud (EC2)**: Amazon EC2 provides on-demand, scalable computing capacity in the Amazon Web Services (AWS) Cloud. |
| 5 | +The containers are publicly maintained, updated, and released periodically by Hugging Face and the AWS team, and are available to all AWS customers in the [Amazon Elastic Container Registry (ECR)](https://aws.github.io/deep-learning-containers/reference/available_images/#huggingface-vllm-inference). You can use them in **Amazon SageMaker AI**: a fully managed platform to build, train, and deploy ML models into a production-ready hosted environment. |
11 | 6 |
|
12 | | -Hugging Face DLCs are open source and licensed under Apache 2.0. Feel free to reach out on our [community forum](https://discuss.huggingface.co/c/sagemaker/17) if you have any questions. |
| 7 | +Hugging Face DLCs are open source and licensed under Apache 2.0. Browse the full list of images and versions on the [Available DLCs](https://huggingface.co/docs/sagemaker/dlcs/available) page, and feel free to reach out on our [community forum](https://discuss.huggingface.co/c/sagemaker/17) if you have any questions. |
13 | 8 |
|
14 | 9 | ## Features & benefits |
15 | 10 |
|
16 | | -The Hugging Face DLCs provide ready-to-use, tested environments to train and deploy Hugging Face models. |
| 11 | +Hugging Face DLCs provide ready-to-use, tested environments to train and deploy Hugging Face models. |
17 | 12 |
|
18 | 13 | ### One command is all you need |
19 | 14 |
|
20 | | -With the new Hugging Face DLCs, train and deploy cutting-edge Transformers-based NLP models in a single line of code. The Hugging Face PyTorch DLCs for training come with all the libraries installed to run a single command e.g. via [TRL CLI](https://huggingface.co/docs/trl/en/clis) to fine-tune LLMs on any setting, either single-GPU, single-node multi-GPU, and more. |
| 15 | +Train and deploy cutting-edge Transformers models in a single line of code. The Hugging Face PyTorch DLCs for training ship with everything needed to run a single command — for example the [TRL CLI](https://huggingface.co/docs/trl/en/clis) — to fine-tune LLMs in any setting, from single-GPU to multi-node multi-GPU. |
21 | 16 |
|
22 | | -### Accelerate machine learning from science to production |
| 17 | +### From science to production |
23 | 18 |
|
24 | | -In addition to Hugging Face DLCs, we created a first-class Hugging Face library for inference, [`sagemaker-huggingface-inference-toolkit`](https://github.com/aws/sagemaker-huggingface-inference-toolkit/tree/main/src/sagemaker_huggingface_inference_toolkit), that comes with the Hugging Face PyTorch DLCs for inference, with full support on serving any PyTorch model on AWS. |
| 19 | +For inference, the general-purpose Hugging Face PyTorch DLC comes with the [`sagemaker-huggingface-inference-toolkit`](https://github.com/aws/sagemaker-huggingface-inference-toolkit), which supports serving any PyTorch model on AWS. Deploy your own trained models or pick from the ever-growing catalog of models on the [Hugging Face Hub](https://huggingface.co/models) with just one more line of code. |
25 | 20 |
|
26 | | -Deploy your trained models for inference with just one more line of code or select any of the ever growing publicly available models from the model Hub. |
| 21 | +### High-performance text generation |
27 | 22 |
|
28 | | -### High-performance text generation and embedding |
| 23 | +For deploying large language models in production, Hugging Face provides dedicated DLCs built around leading open-source inference engines: |
29 | 24 |
|
30 | | -Besides the PyTorch-oriented DLCs, Hugging Face also provides high-performance inference for both text generation and embedding models via the Hugging Face DLCs for both Text Generation Inference (TGI) and Text Embeddings Inference (TEI), respectively. |
| 25 | +* **[vLLM](https://docs.vllm.ai/)** — high-throughput, memory-efficient LLM serving, available for both GPU and AWS AI chips (Neuron). |
| 26 | +* **[SGLang](https://docs.sglang.ai/)** — fast serving with an efficient runtime, available for GPU. |
| 27 | +* **[llama.cpp](https://github.com/ggml-org/llama.cpp)** — lightweight serving of GGUF / quantized models, available for both CPU and GPU. |
31 | 28 |
|
32 | | -The Hugging Face DLC for TGI enables you to deploy any of the +225,000 text generation inference supported models from the Hugging Face Hub, or any custom model as long as its architecture is supported within TGI. |
| 29 | +These engines serve the vast majority of text generation architectures available on the Hugging Face Hub, expose OpenAI-compatible APIs, and support loading models directly from Amazon S3 with no extra configuration. Pick the latest image for your engine and accelerator on the [Available DLCs](https://huggingface.co/docs/sagemaker/dlcs/available) page. |
33 | 30 |
|
34 | | -The Hugging Face DLC for TEI enables you to deploy any of the +12,000 embedding, re-ranking or sequence classification supported models from the Hugging Face Hub, or any custom model as long as its architecture is supported within TEI. |
| 31 | +### High-performance embeddings |
35 | 32 |
|
36 | | -Additionally, these DLCs come with full support for AWS meaning that deploying models from Amazon Simple Storage Service (S3) is also straight forward and requires no configuration. |
| 33 | +For embedding, re-ranking, and sequence-classification workloads, the [Text Embeddings Inference (TEI)](https://huggingface.co/docs/text-embeddings-inference) DLC provides high-performance serving on both CPU and GPU. It can deploy any of the thousands of [supported embedding models](https://huggingface.co/models?other=text-embeddings-inference) on the Hub, or any custom model whose architecture is supported by TEI. |
37 | 34 |
|
38 | 35 | ### Built-in performance |
39 | 36 |
|
40 | | -Hugging Face DLCs feature built-in performance optimizations for PyTorch to train models faster. The DLCs also give you the flexibility to choose a training infrastructure that best aligns with the price/performance ratio for your workload. |
41 | | - |
42 | | -Hugging Face Inference DLCs provide you with production-ready endpoints that scale quickly with your AWS environment, built-in monitoring, and a ton of enterprise features. |
| 37 | +Hugging Face DLCs feature built-in optimizations that let you train faster and serve efficiently, while giving you the flexibility to choose the infrastructure that best fits your price/performance target. The inference DLCs provide production-ready endpoints that scale with your AWS environment, with built-in monitoring and enterprise features. |
0 commit comments