|
1 | 1 | --- |
2 | | -title: Understand the llama.cpp |
| 2 | +title: Understand llama.cpp |
3 | 3 | weight: 3 |
4 | 4 |
|
5 | 5 | ### FIXED, DO NOT MODIFY |
6 | 6 | layout: learningpathall |
7 | 7 | --- |
8 | 8 |
|
9 | | -## Understand the llama.cpp |
| 9 | +## Understand llama.cpp |
10 | 10 |
|
11 | | -**llama.cpp** is an open-source LLM framework implemented in C++ that supports both training and inference. |
12 | | -This learning path focuses only on **inference on the CPU**. |
| 11 | +llama.cpp is an open-source LLM framework implemented in C++ that supports both training and inference. |
13 | 12 |
|
14 | | -The **llama-cli** tool provides a command-line interface to run LLMs with the llama.cpp inference engine. |
| 13 | +This Learning Path focuses on inference on Arm CPUs. |
| 14 | + |
| 15 | +The `llama-cli` tool provides a command-line interface to run LLMs with the llama.cpp inference engine. |
15 | 16 | It supports text generation, chat mode, and grammar-constrained output directly from the terminal. |
16 | 17 |
|
17 | 18 |  |
18 | 19 |
|
19 | | -### What llama-cli does |
20 | | -- Load and interpret LLMs in **.gguf** format |
21 | | -- Build a **compute graph** based on the model structure |
22 | | - - The graph can be divided into subgraphs, each assigned to the most suitable backend device |
23 | | - - In this guide, all operators are executed on the **CPU backend** |
24 | | -- Allocate memory for tensor nodes using the **graph planner** |
25 | | -- Execute tensor nodes in the graph during the **graph_compute** stage, which traverses nodes and forwards work to backend devices |
| 20 | +### What does the Llama CLI do? |
| 21 | + |
| 22 | +Here are the steps performed by `llama-cli`: |
| 23 | + |
| 24 | +1. Load and interpret LLMs in GGUF format |
| 25 | + |
| 26 | +2. Build a compute graph based on the model structure |
| 27 | + |
| 28 | + The graph can be divided into subgraphs, each assigned to the most suitable backend device, but in this Learning Path all operations are executed on the Arm CPU backend. |
| 29 | + |
| 30 | +3. Allocate memory for tensor nodes using the graph planner |
26 | 31 |
|
27 | | -Step2 to Step4 are wrapped inside the function **`llama_decode`**. |
28 | | -During **Prefill** and **Decode**, `llama-cli` repeatedly calls `llama_decode` to generate tokens. |
29 | | -The parameter **`llama_batch`** passed to `llama_decode` differs between stages, containing input tokens, their count, and their positions. |
| 32 | +4. Execute tensor nodes in the graph during the `graph_compute` stage, which traverses nodes and forwards work to backend devices |
| 33 | + |
| 34 | +Steps 2 to 4 are wrapped inside the function `llama_decode`. |
| 35 | +During Prefill and Decode, `llama-cli` repeatedly calls `llama_decode` to generate tokens. |
| 36 | + |
| 37 | +The parameter `llama_batch` passed to `llama_decode` differs between stages, containing input tokens, their count, and their positions. |
| 38 | + |
| 39 | +### What are the components of llama.cpp? |
30 | 40 |
|
31 | | -### Components of llama.cpp |
32 | 41 | The components of llama.cpp include: |
33 | | - |
34 | 42 |
|
35 | | -llama.cpp supports various backends such as `CPU`, `GPU`, `CUDA`, `OpenCL` etc. |
| 43 | + |
| 44 | + |
| 45 | +llama.cpp supports various backends such as `CPU`, `GPU`, `CUDA`, and `OpenCL`. |
| 46 | + |
| 47 | +For the CPU backend, it provides an optimized `ggml-cpu` library, mainly utilizing CPU vector instructions. |
| 48 | + |
| 49 | +For Arm CPUs, the `ggml-cpu` library also offers an `aarch64` trait that leverages 8-bit integer multiply (i8mm) instructions for acceleration. |
36 | 50 |
|
37 | | -For the CPU backend, it provides an optimized `ggml-cpu` library (mainly utilizing CPU vector instructions). |
38 | | -For Arm CPUs, the `ggml-cpu` library also offers an `aarch64` trait that leverages the new **I8MM** instructions for acceleration. |
39 | 51 | The `ggml-cpu` library also integrates the Arm [KleidiAI](https://github.com/ARM-software/kleidiai) library as an additional trait. |
40 | 52 |
|
41 | 53 | ### Prefill and Decode in autoregressive LLMs |
42 | | -Most autoregressive LLMs are Decoder-only model. |
43 | | -Here is a brief introduction to Prefill and Decode stage of autoregressive LLMs. |
44 | | - |
| 54 | + |
| 55 | +An autoregressive LLM is a type of Large Language Model that generates text by predicting the next token (word or word piece) in a sequence based on all the previously generated tokens. |
| 56 | + |
| 57 | +The term "autoregressive" means the model uses its own previous outputs as inputs for generating subsequent outputs, creating a sequential generation process. |
| 58 | + |
| 59 | +For example, when generating the sentence "The cat sat on the", an autoregressive LLM: |
| 60 | +1. Takes the input prompt as context |
| 61 | +2. Predicts the next most likely token (e.g., "mat") |
| 62 | +3. Uses the entire sequence including "mat" to predict the following token |
| 63 | +4. Continues this process token by token until completion |
| 64 | + |
| 65 | +This sequential nature is why autoregressive LLMs have two distinct computational phases: Prefill (processing the initial prompt) and Decode (generating tokens one by one). |
| 66 | + |
| 67 | +Most autoregressive LLMs are Decoder-only models. This refers to the transformer architecture they use, which consists only of decoder blocks from the original Transformer paper. The alternatives to decoder-only models include encoder-only models used for tasks like classification and encoder-decoder models used for tasks like translation. |
| 68 | + |
| 69 | +Decoder-only models like LLaMA have become dominant for text generation because they are simpler to train at scale, can handle both understanding and generation tasks, and are more efficient for text generation. |
| 70 | + |
| 71 | +Here is a brief introduction to Prefill and Decode stages of autoregressive LLMs. |
| 72 | + |
45 | 73 |
|
46 | 74 | At the Prefill stage, multiple input tokens of the prompt are processed. |
47 | | -It mainly performs GEMM (A matrix is multiplied by another matrix) operations to generate the first output token. |
| 75 | + |
| 76 | +It mainly performs GEMM (a matrix is multiplied by another matrix) operations to generate the first output token. |
| 77 | + |
48 | 78 |  |
49 | 79 |
|
50 | | -At the Decode stage, by utilizing the [KV cache](https://huggingface.co/blog/not-lain/kv-caching), it mainly performs GEMV (A vector is multiplied by a matrix) operations to generate subsequent output tokens one by one. |
| 80 | +At the Decode stage, by utilizing the [KV cache](https://huggingface.co/blog/not-lain/kv-caching), it mainly performs GEMV (a vector is multiplied by a matrix) operations to generate subsequent output tokens one by one. |
| 81 | + |
51 | 82 |  |
52 | 83 |
|
53 | | -Therefore, |
54 | | -- **Prefill** is **compute-bound**, dominated by large GEMM operations |
55 | | -- **Decode** is **memory-bound**, dominated by KV cache access and GEMV operations |
| 84 | +In summary, Prefill is compute-bound, dominated by large GEMM operations and Decode is memory-bound, dominated by KV cache access and GEMV operations. |
56 | 85 |
|
57 | | -This can be seen in the subsequent analysis with Streamline. |
| 86 | +You will see this highlighted during the analysis with Streamline. |
0 commit comments