|
| 1 | +--- |
| 2 | +title: Run inference with vLLM |
| 3 | +weight: 4 |
| 4 | + |
| 5 | +### FIXED, DO NOT MODIFY |
| 6 | +layout: learningpathall |
| 7 | +--- |
| 8 | + |
| 9 | +## Run inference on LLama3.1-8B |
| 10 | + |
| 11 | +We will use vLLM to serve an openAI-compatible API that we can use to run inference on Llama3.1-8B. This will demonstrate that the local environment is setup correctly. |
| 12 | + |
| 13 | +Start vLLM’s OpenAI-compatible API server using Llama3.1-8B: |
| 14 | +```bash |
| 15 | +vllm serve meta-llama/Llama-3.1-8B |
| 16 | +``` |
| 17 | + |
| 18 | +Then we can create a test script that sends a request to the server using the OpenAI library. Copy the below to a file named llama_test.py. |
| 19 | + |
| 20 | +```python |
| 21 | +import time |
| 22 | +from openai import OpenAI |
| 23 | +from transformers import AutoTokenizer |
| 24 | + |
| 25 | +# vLLM's OpenAI-compatible server |
| 26 | +client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") |
| 27 | + |
| 28 | +model = "meta-llama/Llama-3.1-8B" # vllm server model |
| 29 | + |
| 30 | +# Define a chat template for the model |
| 31 | +llama3_template = "{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.first and message['role'] != 'system' %}{{ '<|start_header_id|>system<|end_header_id|>\n\n'+ 'You are a helpful assistant.' + '<|eot_id|>' }}{% endif %}{{ content }}{% endfor %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}" |
| 32 | + |
| 33 | +# Define your prompt |
| 34 | +message = [{"role": "user", "content": "Explain Big O notation with two examples."}] |
| 35 | + |
| 36 | +def run(prompt): |
| 37 | + resp = client.completions.create( |
| 38 | + model=model, |
| 39 | + prompt=prompt, |
| 40 | + max_tokens=128, # The maximum number of tokens that can be generated in the completion |
| 41 | + ) |
| 42 | + return resp.choices[0].text |
| 43 | + |
| 44 | +def main(): |
| 45 | + t0 = time.time() |
| 46 | + |
| 47 | + tokenizer = AutoTokenizer.from_pretrained(model) |
| 48 | + tokenizer.chat_template = llama3_template |
| 49 | + prompt = tokenizer.apply_chat_template(message, tokenize=False) |
| 50 | + result = run(prompt) |
| 51 | + |
| 52 | + print(f"\n=== Output ===\n{result}\n") |
| 53 | + print(f"Batch completed in : {time.time() - t0:.2f}s") |
| 54 | + |
| 55 | +if __name__ == "__main__": |
| 56 | + main() |
| 57 | +``` |
| 58 | + |
| 59 | +Now run the script with: |
| 60 | +```bash |
| 61 | +python llama_test.py |
| 62 | +``` |
| 63 | + |
| 64 | +This will return the text generated by the model from your prompt. In the server logs you can see the throughput measured in tokens per second. |
| 65 | + |
| 66 | +You can do the same for the quantised model. Start the server: |
| 67 | +```bash |
| 68 | +vllm serve RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 |
| 69 | +``` |
| 70 | + |
| 71 | +Update your test script to use the quantised model: |
| 72 | +```python |
| 73 | +model = "RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8" |
| 74 | +``` |
| 75 | + |
| 76 | +Run inference on the quantised model: |
| 77 | +```bash |
| 78 | +python llama_test.py |
| 79 | +``` |
| 80 | + |
| 81 | +You have now run inference using both the non-quantised and quantised Llama3.1-8B models. |
| 82 | + |
| 83 | +## Run inference on Whisper |
| 84 | + |
| 85 | +We will use a similar approach to test our ability to run inference on Whisper models. Install the required vLLM audio library then start vLLM’s OpenAI-compatible API server using Whisper-large-v3: |
| 86 | +```bash |
| 87 | +pip install vllm[audio] |
| 88 | + |
| 89 | +vllm serve openai/whisper-large-v3 |
| 90 | +``` |
| 91 | + |
| 92 | +Then we can create a test script that sends a request with an audio file to the server using the OpenAI library. Copy the below to a file named whisper_test.py. |
| 93 | + |
| 94 | +```python |
| 95 | +import time |
| 96 | +from openai import OpenAI |
| 97 | +from vllm.assets.audio import AudioAsset |
| 98 | + |
| 99 | +# vLLM's OpenAI-compatible server |
| 100 | +client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") |
| 101 | + |
| 102 | +model = "openai/whisper-large-v3" # vllm server model |
| 103 | + |
| 104 | +# You can update the below with an audio file of your choosing |
| 105 | +audio_filepath=str(AudioAsset("winning_call").get_local_path()) |
| 106 | + |
| 107 | +def transcribe_audio(): |
| 108 | + with open(audio_filepath, "rb") as audio: |
| 109 | + transcription = client.audio.transcriptions.create( |
| 110 | + model=model, |
| 111 | + file=audio, |
| 112 | + language="en", |
| 113 | + response_format="json", |
| 114 | + temperature=0.0, |
| 115 | + ) |
| 116 | + return transcription.text |
| 117 | + |
| 118 | +def main(): |
| 119 | + t0 = time.time() |
| 120 | + out = transcribe_audio() |
| 121 | + print(f"\n=== Output ===\n{out}\n") |
| 122 | + print(f"Batch completed in : {time.time() - t0:.2f}s") |
| 123 | + |
| 124 | +if __name__ == "__main__": |
| 125 | + main() |
| 126 | +``` |
| 127 | + |
| 128 | +Now run the script with: |
| 129 | +```bash |
| 130 | +python whisper_test.py |
| 131 | +``` |
| 132 | + |
| 133 | +You can do the same for the quantised model. Start the server: |
| 134 | +```bash |
| 135 | +vllm serve RedHatAI/whisper-large-v3-quantized.w8a8 |
| 136 | +``` |
| 137 | + |
| 138 | +Update your test script to use the quantised model: |
| 139 | +```python |
| 140 | +model = "RedHatAI/whisper-large-v3-quantized.w8a8" |
| 141 | +``` |
| 142 | + |
| 143 | +Run inference on the quantised model: |
| 144 | +```bash |
| 145 | +python whisper_test.py |
| 146 | +``` |
| 147 | + |
| 148 | +You now have the quantised and non-quantised Llama and Whisper models on your local machine. You have installed vLLM and demonstrated you can run inference on your models. Now you can move on to benchmarking the Llama models and compare their performance. |
0 commit comments