You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: .github/skills/learning-path-structure-review/SKILL.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,7 +29,7 @@ Use this skill when a Learning Path needs a structural review. Focus on whether
29
29
- Prefer direct relevance, Arm Learning Paths, required tools, foundation knowledge, and logical next steps.
30
30
- Avoid link piles that pull readers away from the task.
31
31
6. Review recap and transition sections:
32
-
- Include concise recap and forward-looking transition at major instructional boundaries.
32
+
- Include concise recap and forward-looking transition at major instructional boundaries. Do not treat a transition sentence alone as a recap. Note the absence of a transition as a finding.
33
33
- Use `what you've learned` for conceptual sections and `what you've accomplished` for task sections.
34
34
- Avoid repeating earlier content verbatim.
35
35
7. If the Learning Path demonstrates Arm-specific performance features, apply the performance integrity checks.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/vllm/_index.md
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,18 +1,20 @@
1
1
---
2
-
title: Build and Run vLLM on Arm Servers
2
+
title: Build and run vLLM on Arm servers
3
+
description: Build vLLM from source on an Arm Linux server, run batch inference with a Hugging Face model, and expose the model through an OpenAI-compatible API.
3
4
4
5
minutes_to_complete: 45
5
6
6
7
who_is_this_for: This is an introductory topic for software developers and AI engineers interested in learning how to use the vLLM library on Arm servers.
7
8
8
9
learning_objectives:
9
10
- Build vLLM from source on an Arm server.
10
-
- Download a Qwen LLM from Hugging Face.
11
+
- Use a Qwen LLM from Hugging Face.
11
12
- Run local batch inference using vLLM.
12
13
- Create and interact with an OpenAI-compatible server provided by vLLM on your Arm server.
13
14
14
15
prerequisites:
15
-
- An [Arm-based instance](/learning-paths/servers-and-cloud-computing/csp/) from a cloud service provider, or a local Arm Linux computer with at least 8 CPUs and 16 GB RAM.
16
+
- An [Arm-based Linux instance](/learning-paths/servers-and-cloud-computing/csp/) from a cloud service provider, or a local Arm Linux computer running Ubuntu 24.04 with at least 8 CPUs, 16 GB RAM, and 50 GB of disk storage.
17
+
- A system that includes support for BFloat16.
16
18
17
19
author: Jason Andrews
18
20
@@ -59,4 +61,3 @@ weight: 1 # _index.md always has weight of 1 to order corr
59
61
layout: "learningpathall"# All files under learning paths have this same wrapper
60
62
learning_path_main_page: "yes"# This should be surfaced when looking for related content. Only set for _index.md of learning path content.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/vllm/vllm-run.md
+15-8Lines changed: 15 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,6 @@
1
1
---
2
2
title: Run batch inference using vLLM
3
+
description: Use vLLM to load a Qwen model from Hugging Face and run batch inference prompts on an Arm server.
3
4
weight: 3
4
5
5
6
### FIXED, DO NOT MODIFY
@@ -8,27 +9,27 @@ layout: learningpathall
8
9
9
10
## Use a model from Hugging Face
10
11
11
-
vLLM is designed to work seamlessly with models from the Hugging Face Hub.
12
+
vLLM is designed to work with models from the Hugging Face Hub.
12
13
13
-
The first time you run vLLM, it downloads the required model. This means that you do not have to explicitly download any models.
14
+
The first time you run vLLM, it downloads the required model. You don't have to explicitly download any models.
14
15
15
-
If you want to use a model that requires you to request access or accept the terms, you need to log in to Hugging Face using a token.
16
+
To use a model that requires you to request access or accept terms and conditions, log in to Hugging Face using a token:
16
17
17
18
```bash
18
19
huggingface-cli login
19
20
```
20
21
21
22
Enter your Hugging Face token. You can generate a token from [Hugging Face Hub](https://huggingface.co/) by clicking your profile on the top right corner and selecting **Access Tokens**.
22
23
23
-
You also need to visit the Hugging Face link printed in the login output and accept the terms by clicking the **Agree and access repository** button or filling out the request-for-access form, depending on the model.
24
+
Visit the Hugging Face link printed in the login output and accept the terms and conditions. Click the **Agree and access repository** button or fill out the request-for-access form, depending on the model.
24
25
25
-
To run batched inference without the need for a login, you can use the `Qwen/Qwen2.5-0.5B-Instruct` model.
26
+
To run batched inference without logging in, use the `Qwen/Qwen2.5-0.5B-Instruct` model.
26
27
27
28
## Create a batch script
28
29
29
-
To run inference with multiple prompts, you can create a simple Python script to load a model and run the prompts.
30
+
To run inference with multiple prompts, create a Python script to load a model and run the prompts.
30
31
31
-
Use a text editor to save the Python script below in a file called `batch.py`:
32
+
Use a text editor to save the following Python script in a file called `batch.py`:
You can try with other prompts and models such as `meta-llama/Llama-3.2-1B`. Continue to learn how to set up an OpenAI-compatible server.
141
+
## What you've accomplished and what's next
142
+
143
+
You've now created a Python batch inference script that loads the `Qwen/Qwen2.5-0.5B-Instruct` model from Hugging Face, configures `bfloat16` precision, and sends multiple prompts to vLLM.
144
+
145
+
You ran the script and confirmed that vLLM starts on the CPU backend, loads the model, processes the prompts, and returns generated text.
146
+
147
+
Next, you'll set up an OpenAI-compatible server so client applications can send requests to vLLM.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/vllm/vllm-server.md
+16-9Lines changed: 16 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,28 +1,31 @@
1
1
---
2
-
title: Run an OpenAI-compatible server
2
+
title: Run an OpenAI-compatible vLLM server
3
+
description: Start a local vLLM OpenAI-compatible server on Arm Linux and send a chat completion request with curl.
3
4
weight: 4
4
5
5
6
### FIXED, DO NOT MODIFY
6
7
layout: learningpathall
7
8
---
8
9
9
-
Instead of a batch run from Python, you can create an OpenAI-compatible server. This allows you to leverage the power of Large Language Models without relying on external APIs.
10
+
## Create a local vLLM server compatible with OpenAI
11
+
12
+
To run Large Language Models (LLMs) without relying on external APIs, create an OpenAI-compatible server.
10
13
11
14
Running a local LLM offers several advantages:
12
15
13
-
* Cost-effective - it avoids the costs associated with using external APIs, especially for high-usage scenarios.
16
+
* Cost-effectiveness - it avoids the costs associated with using external APIs, especially for high-usage scenarios.
14
17
* Privacy - it keeps your data and prompts within your local environment, which enhances privacy and security.
15
-
* Offline Capability - it enables operation without an internet connection, making it ideal for scenarios with limited or unreliable network access.
18
+
* Offline capability - it enables operation without an internet connection, making it ideal for scenarios with limited or unreliable network access.
16
19
17
-
OpenAI compatibility means that you can reuse existing software which was designed to communicate with OpenAI and use it to communicate with your local vLLM service.
20
+
OpenAI compatibility means that you can reuse existing software to communicate with your local vLLM service.
18
21
19
22
Run vLLM with the same `Qwen/Qwen2.5-0.5B-Instruct` model:
The server processes the request, and the output prints the results:
78
+
The server processes the request, and the output is similar to:
76
79
77
80
```output
78
81
"id":"chatcmpl-6677cb4263b34d18b436b9cb8c6a5a65","object":"chat.completion","created":1734044182,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Certainly! Here is a simple \"Hello, World!\" program in C:\n\n```c\n#include <stdio.h>\n\nint main() {\n printf(\"Hello, World!\\n\");\n return 0;\n}\n```\n\nThis program defines a function called `main` which contains the body of the program. Inside the `main` function, it calls the `printf` function to display the text \"Hello, World!\" to the console. The `return 0` statement indicates that the program was successful and the program has ended.\n\nTo compile and run this program:\n\n1. Save the code above to a file named `hello.c`.\n2. Open a terminal or command prompt.\n3. Navigate to the directory where you saved the file.\n4. Compile the program using the following command:\n ```\n gcc hello.c -o hello\n ```\n5. Run the compiled program using the following command:\n ```\n ./hello\n ```\n Or simply type `hello` in the terminal.\n\nYou should see the output:\n\n```\nHello, World!\n```","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":26,"total_tokens":241,"completion_tokens":215,"prompt_tokens_details":null},"prompt_logprobs":null}
79
82
```
80
83
81
-
There are many other experiments you can try. Most Hugging Face models have a **Use this model** button on the top-right of the model card with the instructions for vLLM. You can now use these instructions on your Arm Linux computer.
84
+
## What you've accomplished
85
+
86
+
You've now set up a local OpenAI-compatible server and tested sending requests to the server.
87
+
88
+
You can use the instructions in this Learning Path to experiment with other models on your Arm Linux computer. Most Hugging Face models include a **Use this model** button with instructions for vLLM.
82
89
83
90
You can also try out OpenAI-compatible chat clients to connect to the served model.
description: Prepare an Arm Ubuntu server for vLLM by checking BFloat16 support, installing dependencies, and building the CPU backend from source.
3
4
weight: 2
4
5
5
6
### FIXED, DO NOT MODIFY
@@ -8,35 +9,33 @@ layout: learningpathall
8
9
9
10
## Before you begin
10
11
11
-
To follow the instructions for this Learning Path, you will need an Arm server running Ubuntu 24.04 LTS with at least 8 cores, 16GB of RAM, and 50GB of disk storage. You also need a system which supports BFloat16.
12
+
[Virtual Large Language Model (vLLM)](https://github.com/vllm-project/vllm) is a fast and easy-to-use library for inference and model serving.
12
13
13
-
To check if your system includes BFloat16, use the `lscpu` command:
14
+
You can use vLLM in batch mode, or by running an OpenAI-compatible server.
15
+
16
+
In this Learning Path, you'll learn how to build vLLM from source and run inference on an Arm-based server.
17
+
18
+
Start by checking if your system includes BFloat16 using the `lscpu` command:
14
19
15
20
```console
16
21
lscpu | grep bf16
17
22
```
18
23
19
-
If the `Flags` are printed, you have a processor with BFloat16.
24
+
If you have a processor with BFloat16, the output is similar to:
If the result is blank, you do not have a processor with BFloat16.
30
+
If the output is blank, you don't have a processor with BFloat16.
26
31
27
-
BFloat16 provides improved performance and smaller memory footprint with the same dynamic range. You might experience a drop in model inference accuracy with BFloat16, but the impact is acceptable for the majority of applications.
32
+
BFloat16 provides improved performance and smaller memory footprint with the same dynamic range. You might experience a drop in model inference accuracy with BFloat16, but the tradeoff is acceptable for the majority of applications.
28
33
29
-
The instructions have been tested on an AWS Graviton3 `m7g.2xlarge` instance.
34
+
The instructions in this Learning Path have been tested on an AWS Graviton3 `m7g.2xlarge` instance.
30
35
31
-
## What is vLLM?
36
+
## Install dependencies to build vLLM
32
37
33
-
[vLLM](https://github.com/vllm-project/vllm) stands for Virtual Large Language Model, and is a fast and easy-to-use library for inference and model serving.
34
-
35
-
You can use vLLM in batch mode, or by running an OpenAI-compatible server.
36
-
37
-
In this Learning Path, you will learn how to build vLLM from source and run inference on an Arm-based server, highlighting its effectiveness.
38
-
39
-
### What software do I need to install to build vLLM?
38
+
After validating that your system supports BFloat16, install vLLM dependencies.
40
39
41
40
First, ensure your system is up-to-date and install the required tools and libraries:
Four environment variables are required. You can enter these at the command line or add them to your `$HOME/.bashrc` file and source the file.
54
+
Next, set up required environment variables. You can either enter the variables at the command line or add them to your `$HOME/.bashrc` file and source the file.
56
55
57
-
To add them at the command line, use the following:
56
+
To add the environment variables at the command line, run:
58
57
59
58
```bash
60
59
export CCACHE_DIR=/home/ubuntu/.cache/ccache
@@ -72,14 +71,14 @@ source env/bin/activate
72
71
73
72
Your command-line prompt is prefixed by `(env)`, which indicates that you are in the Python virtual environment.
74
73
75
-
Now update Pip and install Python packages:
74
+
Now update `pip` and install Python packages:
76
75
77
76
```bash
78
77
pip install --upgrade pip
79
78
pip install py-cpuinfo
80
79
```
81
80
82
-
### How do I download vLLM and build it?
81
+
##Download and build vLLM
83
82
84
83
First, clone the vLLM repository from GitHub:
85
84
@@ -90,9 +89,9 @@ git checkout releases/v0.11.0
90
89
```
91
90
92
91
{{% notice Note %}}
93
-
The Git checkout specifies a specific hash known to work for this example.
92
+
The Git checkout specifies a version known to work for this example.
94
93
95
-
Omit this command to use the latest code on the main branch.
94
+
Omit this checkout command to use the latest code on the main branch.
You are now ready to download a large language model (LLM) and run vLLM.
118
+
## What you've accomplished and what's next
119
+
120
+
You've now verified BFloat16 support, installed the build dependencies, configured the required environment variables, and built vLLM from source for the CPU backend.
121
+
122
+
Next, you'll use vLLM with a Hugging Face model to run batch inference on your Arm server.
0 commit comments