|
1 | 1 | --- |
2 | | -title: Overview |
| 2 | +title: Overview of AFM-4.5B deployment on Google Cloud Axion with Llama.cpp |
3 | 3 | weight: 2 |
4 | 4 |
|
5 | 5 | ### FIXED, DO NOT MODIFY |
6 | 6 | layout: learningpathall |
7 | 7 | --- |
8 | 8 |
|
9 | | -## The AFM-4.5B model |
| 9 | +## AFM-4.5B model and deployment workflow |
10 | 10 |
|
11 | 11 | [AFM-4.5B](https://huggingface.co/arcee-ai/AFM-4.5B) is a 4.5-billion-parameter foundation model designed to balance accuracy, efficiency, and broad language coverage. Trained on nearly 8 trillion tokens of carefully filtered data, it performs well across a wide range of languages, including Arabic, English, French, German, Hindi, Italian, Korean, Mandarin, Portuguese, Russian, and Spanish. |
12 | 12 |
|
13 | | -In this Learning Path, you'll deploy [AFM-4.5B](https://huggingface.co/arcee-ai/AFM-4.5B) using [Llama.cpp](https://github.com/ggerganov/llama.cpp) on an Arm-based Google Cloud Axion instance. You’ll walk through the full workflow, from setting up your environment and compiling the runtime, to downloading, quantizing, and running inference on the model. You'll also evaluate model quality using perplexity, a common metric for measuring how well a language model predicts text. |
| 13 | +In this Learning Path, you’ll deploy [AFM-4.5B](https://huggingface.co/arcee-ai/AFM-4.5B) using [Llama.cpp](https://github.com/ggerganov/llama.cpp) on a Google Cloud Axion Arm64 instance. You’ll walk through the full workflow, from setting up your environment and compiling the runtime, to downloading, quantizing, and running inference on the model. You’ll also evaluate model quality using perplexity, a standard metric for how well a language model predicts text. |
14 | 14 |
|
15 | 15 | This hands-on guide helps developers build cost-efficient, high-performance LLM applications on modern Arm server infrastructure using open-source tools and real-world deployment practices. |
16 | 16 |
|
17 | | -### LLM deployment workflow on Google Axion |
| 17 | +### Deployment workflow for AFM-4.5B on Google Cloud Axion |
18 | 18 |
|
19 | | -- **Provision compute**: launch a Google Cloud instance using an Axion-based instance type (for example, `c4a-standard-16`) |
| 19 | +- **Provision compute**: launch a Google Cloud instance using an Axion-based instance type (for example, `c4a-standard-16`) |
| 20 | +- **Set up your environment**: install build tools and dependencies (CMake, Python, Git) |
| 21 | +- **Build the inference engine**: clone the [Llama.cpp](https://github.com/ggerganov/llama.cpp) repository and compile the project for your Arm-based environment |
| 22 | +- **Prepare the model**: download the AFM-4.5B model files from Hugging Face and use Llama.cpp’s quantization tools to reduce model size and optimize performance |
| 23 | +- **Run inference**: load the quantized model and run sample prompts using Llama.cpp |
| 24 | +- **Evaluate model quality**: calculate perplexity or use other metrics to assess performance |
20 | 25 |
|
21 | | -- **Set up your environment**: install the required build tools and dependencies (such as CMake, Python, and Git) |
22 | | - |
23 | | -- **Build the inference engine**: clone the [Llama.cpp](https://github.com/ggerganov/llama.cpp) repository and compile the project for your Arm-based environment |
24 | | - |
25 | | -- **Prepare the model**: download the **AFM-4.5B** model files from Hugging Face and use Llama.cpp's quantization tools to reduce model size and optimize performance |
26 | | - |
27 | | -- **Run inference**: load the quantized model and run sample prompts using Llama.cpp. |
28 | | - |
29 | | -- **Evaluate model quality**: calculate **perplexity** or use other metrics to assess model performance |
30 | | - |
31 | | -{{< notice Note>}} |
| 26 | +{{< notice Note >}} |
32 | 27 | You can reuse this deployment flow with other models supported by Llama.cpp by swapping out the model file and adjusting quantization settings. |
33 | 28 | {{< /notice >}} |
34 | | - |
35 | | - |
36 | | - |
37 | | - |
0 commit comments