Skip to content

Commit e52befd

Browse files
dwarezdwarez
authored
Upgrading Sagemaker docs and tutorials to sagemaker-sdk v3 (#2556)
* change: deploy-embedding-models notebook aligned to sagemaker sdk v3 * change: upgraded fine-tune-embedding-models notebook to sagemaker-sdk v3 fix: removed unecessary dependencies from the `run_mnr.py` script because now we're using an image that already carries most of them change: asset folder renamed to scripts because it only contains a script and a requirements file (furthermore the notebook was pointing to the scrips/ folder and not to assets/) Signed-off-by: DWarez <dario.salvati@huggingface.co> * change: aligned evaluate-llm-lighteval with sagemaker-sdk v3 change: upgraded packages involved change: added requirements and launch script directly in a scripts/ folder, removed the instructions to write them at runtime Signed-off-by: DWarez <dario.salvati@huggingface.co> * change: deploy-llama-3-3-70b-inferentia2 aligned on sdk v3 Signed-off-by: DWarez <dario.salvati@huggingface.co> * fix: cloning llm-lab repository to grab scripts for deepseek-ocr-sagemaker Signed-off-by: DWarez <dario.salvati@huggingface.co> * change: deploy-sagemaker-sdk docs aligned for sdk v3 Signed-off-by: DWarez <dario.salvati@huggingface.co> * change: align all sagemaker-sdk tutorial docs to sdk v3 Signed-off-by: DWarez <dario.salvati@huggingface.co> * change: align jumpstart tutorial to sdk v3 Signed-off-by: DWarez <dario.salvati@huggingface.co> * fix: index Signed-off-by: DWarez <dario.salvati@huggingface.co> * add: sdk v3 banner in fine-tune-trl-cli example Signed-off-by: DWarez <dario.salvati@huggingface.co> --------- Signed-off-by: DWarez <dario.salvati@huggingface.co>
1 parent df077a1 commit e52befd

19 files changed

Lines changed: 1198 additions & 1615 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,4 @@ __pycache__/
44
.idea/
55

66
.DS_Store
7+
.ipynb_checkpoints/

‎docs/sagemaker/notebooks/sagemaker-sdk/deepseek-ocr-sagemaker/sagemaker-notebook.ipynb‎

Lines changed: 70 additions & 905 deletions
Large diffs are not rendered by default.

‎docs/sagemaker/notebooks/sagemaker-sdk/deploy-embedding-models/sagemaker-notebook.ipynb‎

Lines changed: 80 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -53,15 +53,15 @@
5353
"metadata": {},
5454
"outputs": [],
5555
"source": [
56-
"!pip install \"sagemaker<3.0.0\" --upgrade --quiet"
56+
"!pip install \"sagemaker>=3.0.0\" --upgrade --quiet"
5757
]
5858
},
5959
{
6060
"cell_type": "markdown",
6161
"metadata": {},
6262
"source": [
63-
"> [!WARNING]\n",
64-
"> [SageMaker Python SDK v3 has been recently released](https://github.com/aws/sagemaker-python-sdk), so unless specified otherwise, all the documentation and tutorials are still using the [SageMaker Python SDK v2](https://github.com/aws/sagemaker-python-sdk/tree/master-v2). We are actively working on updating all the tutorials and examples, but in the meantime make sure to install the SageMaker SDK as `pip install \"sagemaker<3.0.0\"`."
63+
"> [!NOTE]\n",
64+
"> This example uses the [SageMaker Python SDK v3](https://github.com/aws/sagemaker-python-sdk). v3 introduces a new, framework-agnostic API built around `ModelBuilder` (inference) and `ModelTrainer` (training), which replaces the v2 `HuggingFaceModel` and `HuggingFace` classes."
6565
]
6666
},
6767
{
@@ -78,26 +78,22 @@
7878
"metadata": {},
7979
"outputs": [],
8080
"source": [
81-
"import sagemaker\n",
8281
"import boto3\n",
83-
"sess = sagemaker.Session()\n",
82+
"from sagemaker.core.helper.session_helper import Session, get_execution_role\n",
83+
"\n",
84+
"sess = Session()\n",
8485
"# sagemaker session bucket -> used for uploading data, models and logs\n",
85-
"# sagemaker will automatically create this bucket if it not exists\n",
86-
"sagemaker_session_bucket=None\n",
87-
"if sagemaker_session_bucket is None and sess is not None:\n",
88-
" # set to default bucket if a bucket name is not given\n",
89-
" sagemaker_session_bucket = sess.default_bucket()\n",
86+
"# sagemaker will automatically create this bucket if it does not exist\n",
87+
"sagemaker_session_bucket = sess.default_bucket()\n",
9088
"\n",
9189
"try:\n",
92-
" role = sagemaker.get_execution_role()\n",
93-
"except ValueError:\n",
90+
" role = get_execution_role()\n",
91+
"except Exception:\n",
9492
" iam = boto3.client('iam')\n",
9593
" role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n",
9694
"\n",
97-
"sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n",
98-
"\n",
9995
"print(f\"sagemaker role arn: {role}\")\n",
100-
"print(f\"sagemaker session region: {sess.boto_region_name}\")\n"
96+
"print(f\"sagemaker session region: {sess.boto_region_name}\")"
10197
]
10298
},
10399
{
@@ -107,7 +103,7 @@
107103
"source": [
108104
"## 2. Retrieve the new Hugging Face Embedding Container\n",
109105
"\n",
110-
"Compared to deploying regular Hugging Face models we first need to retrieve the container uri and provide it to our `HuggingFaceModel` model class with a `image_uri` pointing to the image. To retrieve the new Hugging Face Embedding Container in Amazon SageMaker, we can use the `get_huggingface_llm_image_uri` method provided by the `sagemaker` SDK. This method allows us to retrieve the URI for the desired Hugging Face Embedding Container. Important to note is that TEI has 2 different versions for cpu and gpu, so we create a helper function to retrieve the correct image uri based on the instance type. "
106+
"To deploy an embedding model we provide the TEI container image to a `ModelBuilder`. To retrieve the URI of the Hugging Face Embedding Container in Amazon SageMaker, we use the `image_uris.retrieve` helper from `sagemaker.core`. TEI has two different versions for CPU and GPU, so we create a helper function to retrieve the correct image URI based on the instance type. The processor (CPU/GPU) is inferred from the `instance_type`."
111107
]
112108
},
113109
{
@@ -116,12 +112,18 @@
116112
"metadata": {},
117113
"outputs": [],
118114
"source": [
119-
"from sagemaker.huggingface import get_huggingface_llm_image_uri\n",
115+
"from sagemaker.core import image_uris\n",
120116
"\n",
121117
"# retrieve the image uri based on instance type\n",
122118
"def get_image_uri(instance_type):\n",
123-
" key = \"huggingface-tei\" if instance_type.startswith(\"ml.g\") or instance_type.startswith(\"ml.p\") else \"huggingface-tei-cpu\"\n",
124-
" return get_huggingface_llm_image_uri(key, version=\"1.4.0\")\n"
119+
" framework = \"huggingface-tei\" if instance_type.startswith((\"ml.g\", \"ml.p\")) else \"huggingface-tei-cpu\"\n",
120+
" return image_uris.retrieve(\n",
121+
" framework=framework,\n",
122+
" region=sess.boto_region_name,\n",
123+
" version=\"1.8.2\",\n",
124+
" image_scope=\"inference\",\n",
125+
" instance_type=instance_type,\n",
126+
" )"
125127
]
126128
},
127129
{
@@ -131,7 +133,7 @@
131133
"source": [
132134
"## 3. Deploy Snowflake Arctic to Amazon SageMaker\n",
133135
"\n",
134-
"To deploy [Snowflake/snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m) to Amazon SageMaker we create a `HuggingFaceModel` model class and define our endpoint configuration including the `HF_MODEL_ID`, `instance_type` etc. We will use a `c6i.2xlarge` instance type, which has 4 Intel Ice-Lake vCPUs, 8GB of memory and costs around $0.204 per hour. "
136+
"To deploy [Snowflake/snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m) to Amazon SageMaker we create a `ModelBuilder` and define our configuration: the Hugging Face model ID, the TEI container image, the instance type and the model server (`ModelServer.TEI`). `ModelBuilder` automatically sets `HF_MODEL_ID` from the model ID and lets TEI download the model directly from the Hugging Face Hub. We will use a `ml.g5.xlarge` GPU instance type."
135137
]
136138
},
137139
{
@@ -140,31 +142,47 @@
140142
"metadata": {},
141143
"outputs": [],
142144
"source": [
143-
"import json\n",
144-
"from sagemaker.huggingface import HuggingFaceModel\n",
145+
"role = \"arn:aws:iam::754289655784:role/sagemaker_execution_role\""
146+
]
147+
},
148+
{
149+
"cell_type": "code",
150+
"execution_count": null,
151+
"metadata": {},
152+
"outputs": [],
153+
"source": [
154+
"from sagemaker.serve import ModelBuilder, ModelServer\n",
155+
"from sagemaker.serve.builder.schema_builder import SchemaBuilder\n",
145156
"\n",
146157
"# sagemaker config\n",
147158
"instance_type = \"ml.g5.xlarge\"\n",
159+
"model_id = \"Snowflake/snowflake-arctic-embed-m\" # model_id from hf.co/models\n",
148160
"\n",
149-
"# Define Model and Endpoint configuration parameter\n",
150-
"config = {\n",
151-
" 'HF_MODEL_ID': \"Snowflake/snowflake-arctic-embed-m\", # model_id from hf.co/models\n",
152-
"}\n",
161+
"# Create a ModelBuilder for the TEI (Text Embeddings Inference) server.\n",
162+
"# ModelBuilder sets HF_MODEL_ID from `model` and pulls the model from the Hub.\n",
163+
"model_builder = ModelBuilder(\n",
164+
" model=model_id,\n",
165+
" role_arn=role,\n",
166+
" sagemaker_session=sess,\n",
167+
" instance_type=instance_type,\n",
168+
" image_uri=get_image_uri(instance_type),\n",
169+
" model_server=ModelServer.TEI,\n",
170+
" schema_builder=SchemaBuilder(\n",
171+
" sample_input={\"inputs\": \"embed this sentence\"},\n",
172+
" sample_output=[[0.0] * 768],\n",
173+
" ),\n",
174+
")\n",
153175
"\n",
154-
"# create HuggingFaceModel with the image uri\n",
155-
"emb_model = HuggingFaceModel(\n",
156-
" role=role,\n",
157-
" image_uri=get_image_uri(instance_type),\n",
158-
" env=config\n",
159-
")"
176+
"# Build the SageMaker model resource\n",
177+
"emb_model = model_builder.build()"
160178
]
161179
},
162180
{
163181
"attachments": {},
164182
"cell_type": "markdown",
165183
"metadata": {},
166184
"source": [
167-
"After we have created the `HuggingFaceModel` we can deploy it to Amazon SageMaker using the `deploy` method. We will deploy the model with the `ml.c6i.2xlarge` instance type."
185+
"After we have built the model with `ModelBuilder` we can deploy it to Amazon SageMaker using the `deploy` method. We will deploy the model with the `ml.g5.xlarge` instance type. `deploy` returns an `Endpoint` object that we use to run inference."
168186
]
169187
},
170188
{
@@ -173,12 +191,11 @@
173191
"metadata": {},
174192
"outputs": [],
175193
"source": [
176-
"# Deploy model to an endpoint\n",
177-
"# https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.Model.deploy\n",
178-
"emb = emb_model.deploy(\n",
179-
" initial_instance_count=1,\n",
180-
" instance_type=instance_type,\n",
181-
")\n"
194+
"# Deploy the model to a real-time endpoint\n",
195+
"emb = model_builder.deploy(\n",
196+
" initial_instance_count=1,\n",
197+
" instance_type=instance_type,\n",
198+
")"
182199
]
183200
},
184201
{
@@ -196,7 +213,7 @@
196213
"source": [
197214
"## 4. Run and evaluate Inference performance\n",
198215
"\n",
199-
"After our endpoint is deployed we can run inference on it. We will use the `predict` method from the `predictor` to run inference on our endpoint.\n"
216+
"After our endpoint is deployed we can run inference on it. We use the `invoke` method of the `Endpoint` to send requests. The request body is JSON and the response `body` contains the embeddings."
200217
]
201218
},
202219
{
@@ -205,16 +222,18 @@
205222
"metadata": {},
206223
"outputs": [],
207224
"source": [
225+
"import json\n",
226+
"\n",
208227
"data = {\n",
209-
" \"inputs\": \"the mesmerizing performances of the leads keep the film grounded and keep the audience riveted .\",\n",
228+
" \"inputs\": \"the mesmerizing performances of the leads keep the film grounded and keep the audience riveted .\",\n",
210229
"}\n",
211-
" \n",
212-
"res = emb.predict(data=data)\n",
213-
" \n",
214-
" \n",
230+
"\n",
231+
"res = emb.invoke(body=json.dumps(data), content_type=\"application/json\")\n",
232+
"embeddings = json.loads(res.body.read())\n",
233+
"\n",
215234
"# print some results\n",
216-
"print(f\"length of embeddings: {len(res[0])}\")\n",
217-
"print(f\"first 10 elements of embeddings: {res[0][:10]}\")"
235+
"print(f\"length of embeddings: {len(embeddings[0])}\")\n",
236+
"print(f\"first 10 elements of embeddings: {embeddings[0][:10]}\")"
218237
]
219238
},
220239
{
@@ -236,18 +255,21 @@
236255
"source": [
237256
"import threading\n",
238257
"import time\n",
258+
"\n",
239259
"number_of_threads = 10\n",
240260
"number_of_requests = int(3900 // number_of_threads)\n",
241261
"print(f\"number of threads: {number_of_threads}\")\n",
242262
"print(f\"number of requests per thread: {number_of_requests}\")\n",
243-
" \n",
244-
"def send_rquests():\n",
263+
"\n",
264+
"# input counted at https://huggingface.co/spaces/Xenova/the-tokenizer-playground for ~100 tokens\n",
265+
"payload = json.dumps({\"inputs\": \"Hugging Face is a company and a popular platform in the field of natural language processing (NLP) and machine learning. They are known for their contributions to the development of state-of-the-art models for various NLP tasks and for providing a platform that facilitates the sharing and usage of pre-trained models. One of the key offerings from Hugging Face is the Transformers library, which is an open-source library for working with a variety of pre-trained transformer models, including those for text generation, translation, summarization, question answering, and more. The library is widely used in the research and development of NLP applications and is supported by a large and active community. Hugging Face also provides a model hub where users can discover, share, and download pre-trained models. Additionally, they offer tools and frameworks to make it easier for developers to integrate and use these models in their own projects. The company has played a significant role in advancing the field of NLP and making cutting-edge models more accessible to the broader community. Hugging Face also provides a model hub where users can discover, share, and download pre-trained models. Additionally, they offer tools and frameworks to make it easier for developers and ma\"})\n",
266+
"\n",
267+
"def send_requests():\n",
245268
" for _ in range(number_of_requests):\n",
246-
" # input counted at https://huggingface.co/spaces/Xenova/the-tokenizer-playground for 100 tokens\n",
247-
" emb.predict(data={\"inputs\": \"Hugging Face is a company and a popular platform in the field of natural language processing (NLP) and machine learning. They are known for their contributions to the development of state-of-the-art models for various NLP tasks and for providing a platform that facilitates the sharing and usage of pre-trained models. One of the key offerings from Hugging Face is the Transformers library, which is an open-source library for working with a variety of pre-trained transformer models, including those for text generation, translation, summarization, question answering, and more. The library is widely used in the research and development of NLP applications and is supported by a large and active community. Hugging Face also provides a model hub where users can discover, share, and download pre-trained models. Additionally, they offer tools and frameworks to make it easier for developers to integrate and use these models in their own projects. The company has played a significant role in advancing the field of NLP and making cutting-edge models more accessible to the broader community. Hugging Face also provides a model hub where users can discover, share, and download pre-trained models. Additionally, they offer tools and frameworks to make it easier for developers and ma\"})\n",
248-
" \n",
269+
" emb.invoke(body=payload, content_type=\"application/json\")\n",
270+
"\n",
249271
"# Create multiple threads\n",
250-
"threads = [threading.Thread(target=send_rquests) for _ in range(number_of_threads) ]\n",
272+
"threads = [threading.Thread(target=send_requests) for _ in range(number_of_threads)]\n",
251273
"# start all threads\n",
252274
"start = time.time()\n",
253275
"[t.start() for t in threads]\n",
@@ -300,14 +322,14 @@
300322
"metadata": {},
301323
"outputs": [],
302324
"source": [
303-
"emb.delete_model()\n",
304-
"emb.delete_endpoint()"
325+
"emb.delete()\n",
326+
"emb_model.delete()"
305327
]
306328
}
307329
],
308330
"metadata": {
309331
"kernelspec": {
310-
"display_name": "hf",
332+
"display_name": "Python 3 (ipykernel)",
311333
"language": "python",
312334
"name": "python3"
313335
},
@@ -321,15 +343,14 @@
321343
"name": "python",
322344
"nbconvert_exporter": "python",
323345
"pygments_lexer": "ipython3",
324-
"version": "3.9.13"
346+
"version": "3.12.13"
325347
},
326-
"orig_nbformat": 4,
327348
"vscode": {
328349
"interpreter": {
329350
"hash": "5fcf248a74081676ead7e77f54b2c239ba2921b952f7cbcdbbe5427323165924"
330351
}
331352
}
332353
},
333354
"nbformat": 4,
334-
"nbformat_minor": 2
355+
"nbformat_minor": 4
335356
}

0 commit comments

Comments
 (0)