First review of llama.cpp with Streamline Learning Path

Author: jasonrandrews

content/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/1_overview.md

@@ -6,26 +6,21 @@ weight: 2
 layout: learningpathall
 ---
 
-## Overview: Profiling LLMs on Arm CPUs with Streamline
+## Profiling LLMs on Arm CPUs with Streamline
 
-Large Language Models (LLMs) run efficiently on Arm CPUs.  
-Frameworks that run LLMs, such as [**llama.cpp**](https://github.com/ggml-org/llama.cpp), provides a convenient framework for running LLMs, it also comes with a certain level of complexity. 
+Deploying Large Language Models (LLMs) on Arm CPUs provides a power-efficient and flexible solution. While larger models may benefit from GPU acceleration, techniques like quantization enable a wide range of LLMs to perform effectively on CPUs alone.  
+
+Frameworks such as [**llama.cpp**](https://github.com/ggml-org/llama.cpp), provide a convenient way to run LLMs, but it also comes with a certain level of complexity. 
 
 To analyze their execution and use profiling insights for optimization, you need both a basic understanding of transformer architectures and the right analysis tools.
 
-This learning path demonstrates how to use the **llama-cli** application from llama.cpp together with **Arm Streamline** to analyze the efficiency of LLM inference on Arm CPUs.  
+This Learning Path demonstrates how to use `llama-cli` application from llama.cpp together with Arm Streamline to analyze the efficiency of LLM inference on Arm CPUs.  
 
-In this guide you will learn how to:
-- Profile token generation at the **Prefill** and **Decode** stages
+You will learn how to:
+- Profile token generation at the Prefill and Decode stages
 - Profile execution of individual tensor nodes and operators
-- Profile LLM execution across **multiple threads and cores**
+- Profile LLM execution across multiple threads and cores
 
-You will run the **Qwen1_5-0_5b-chat-q4_0.gguf** model with llama-cli on **Arm64 Linux** and use Streamline for analysis.  
-The same method can also be applied to **Arm64 Android** platforms.  
+You will run the `Qwen1_5-0_5b-chat-q4_0.gguf` model using `llama-cli` on Arm Linux and use Streamline for analysis.  
 
-## Prerequisites
-Before starting this guide, you should be familiar with:
-- Basic understanding of llama.cpp
-- Understanding of transformer model
-- Knowledge of Streamline usage
-- An Arm Neoverse or Cortex-A hardware platform running Linux or Android to test the application
+The same method can also be applied to Android platforms.

content/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/2_llama.cpp_intro.md

@@ -1,57 +1,86 @@
 ---
-title: Understand the llama.cpp
+title: Understand llama.cpp
 weight: 3
 
 ### FIXED, DO NOT MODIFY
 layout: learningpathall
 ---
 
-## Understand the llama.cpp
+## Understand llama.cpp
 
-**llama.cpp** is an open-source LLM framework implemented in C++ that supports both training and inference.
-This learning path focuses only on **inference on the CPU**.  
+llama.cpp is an open-source LLM framework implemented in C++ that supports both training and inference.
 
-The **llama-cli** tool provides a command-line interface to run LLMs with the llama.cpp inference engine. 
+This Learning Path focuses on inference on Arm CPUs.  
+
+The `llama-cli` tool provides a command-line interface to run LLMs with the llama.cpp inference engine. 
 It supports text generation, chat mode, and grammar-constrained output directly from the terminal.  
 
 ![text#center](images/llama_structure.png "Figure 1. llama-cli Flow")
 
-### What llama-cli does
-- Load and interpret LLMs in **.gguf** format  
-- Build a **compute graph** based on the model structure  
-  - The graph can be divided into subgraphs, each assigned to the most suitable backend device  
-  - In this guide, all operators are executed on the **CPU backend**  
-- Allocate memory for tensor nodes using the **graph planner**  
-- Execute tensor nodes in the graph during the **graph_compute** stage, which traverses nodes and forwards work to backend devices  
+### What does the Llama CLI do? 
+
+Here are the steps performed by `llama-cli`:
+
+1. Load and interpret LLMs in GGUF format  
+
+2. Build a compute graph based on the model structure  
+
+   The graph can be divided into subgraphs, each assigned to the most suitable backend device, but in this Learning Path all operations are executed on the Arm CPU backend. 
+
+3. Allocate memory for tensor nodes using the graph planner  
 
-Step2 to Step4 are wrapped inside the function **`llama_decode`**.
-During **Prefill** and **Decode**, `llama-cli` repeatedly calls `llama_decode` to generate tokens.  
-The parameter **`llama_batch`** passed to `llama_decode` differs between stages, containing input tokens, their count, and their positions.  
+4. Execute tensor nodes in the graph during the `graph_compute` stage, which traverses nodes and forwards work to backend devices  
+
+Steps 2 to 4 are wrapped inside the function `llama_decode`.
+During Prefill and Decode, `llama-cli` repeatedly calls `llama_decode` to generate tokens.  
+
+The parameter `llama_batch` passed to `llama_decode` differs between stages, containing input tokens, their count, and their positions.  
+
+### What are the components of llama.cpp?
 
-### Components of llama.cpp
 The components of llama.cpp include: 
-![text#center](images/llama_componetns.jpg "Figure 2. llmama.cpp components")
 
-llama.cpp supports various backends such as `CPU`, `GPU`, `CUDA`, `OpenCL` etc.
+![text#center](images/llama_components.jpg "Figure 2. llama.cpp components")
+
+llama.cpp supports various backends such as `CPU`, `GPU`, `CUDA`, and `OpenCL`.
+
+For the CPU backend, it provides an optimized `ggml-cpu` library, mainly utilizing CPU vector instructions. 
+
+For Arm CPUs, the `ggml-cpu` library also offers an `aarch64` trait that leverages 8-bit integer multiply (i8mm) instructions for acceleration. 
 
-For the CPU backend, it provides an optimized `ggml-cpu` library (mainly utilizing CPU vector instructions). 
-For Arm CPUs, the `ggml-cpu` library also offers an `aarch64` trait that leverages the new **I8MM** instructions for acceleration. 
 The `ggml-cpu` library also integrates the Arm [KleidiAI](https://github.com/ARM-software/kleidiai) library as an additional trait.
 
 ### Prefill and Decode in autoregressive LLMs
-Most autoregressive LLMs are Decoder-only model.
-Here is a brief introduction to Prefill and Decode stage of autoregressive LLMs.
-![text#center](images/llm_prefill_decode.jpg "Figure 3. Prefill and Decode stage")
+
+An autoregressive LLM is a type of Large Language Model that generates text by predicting the next token (word or word piece) in a sequence based on all the previously generated tokens. 
+
+The term "autoregressive" means the model uses its own previous outputs as inputs for generating subsequent outputs, creating a sequential generation process.
+
+For example, when generating the sentence "The cat sat on the", an autoregressive LLM:
+1. Takes the input prompt as context
+2. Predicts the next most likely token (e.g., "mat")  
+3. Uses the entire sequence including "mat" to predict the following token
+4. Continues this process token by token until completion
+
+This sequential nature is why autoregressive LLMs have two distinct computational phases: Prefill (processing the initial prompt) and Decode (generating tokens one by one).
+
+Most autoregressive LLMs are Decoder-only models. This refers to the transformer architecture they use, which consists only of decoder blocks from the original Transformer paper. The alternatives to decoder-only models include encoder-only models used for tasks like classification and encoder-decoder models used for tasks like translation. 
+
+Decoder-only models like LLaMA have become dominant for text generation because they are simpler to train at scale, can handle both understanding and generation tasks, and are more efficient for text generation. 
+
+Here is a brief introduction to Prefill and Decode stages of autoregressive LLMs.
+![text#center](images/llm_prefill_decode.jpg "Figure 3. Prefill and Decode stages")
 
 At the Prefill stage, multiple input tokens of the prompt are processed.
-It mainly performs GEMM (A matrix is multiplied by another matrix) operations to generate the first output token. 
+
+It mainly performs GEMM (a matrix is multiplied by another matrix) operations to generate the first output token. 
+
 ![text#center](images/transformer_prefill.jpg "Figure 4. Prefill stage")
 
-At the Decode stage, by utilizing the [KV cache](https://huggingface.co/blog/not-lain/kv-caching), it mainly performs GEMV (A vector is multiplied by a matrix) operations to generate subsequent output tokens one by one.
+At the Decode stage, by utilizing the [KV cache](https://huggingface.co/blog/not-lain/kv-caching), it mainly performs GEMV (a vector is multiplied by a matrix) operations to generate subsequent output tokens one by one.
+
 ![text#center](images/transformer_decode.jpg "Figure 5. Decode stage")
 
-Therefore, 
-- **Prefill** is **compute-bound**, dominated by large GEMM operations  
-- **Decode** is **memory-bound**, dominated by KV cache access and GEMV operations 
+In summary, Prefill is compute-bound, dominated by large GEMM operations and Decode is memory-bound, dominated by KV cache access and GEMV operations. 
 
-This can be seen in the subsequent analysis with Streamline.
+You will see this highlighted during the analysis with Streamline.

content/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/3_llama.cpp_annotation.md

@@ -1,36 +1,40 @@
 ---
-title: Integrating Streamline Annotations into llama.cpp
+title: Integrate Streamline Annotations into llama.cpp
 weight: 4
 
 ### FIXED, DO NOT MODIFY
 layout: learningpathall
 ---
 
-## Integrating Streamline Annotations into llama.cpp
+## Integrate Streamline Annotations into llama.cpp
 
-To visualize token generation at the **Prefill** and **Decode** stages, we use **Streamline’s Annotation Marker** feature.  
-This requires integrating annotation support into the **llama.cpp** project.  
-More information about the Annotation Marker API can be found [here](https://developer.arm.com/documentation/101816/9-7/Annotate-your-code?lang=en).
+To visualize token generation at the Prefill and Decode stages, you can use Streamline's Annotation Marker feature.  
+
+This requires integrating annotation support into the llama.cpp project.  
+
+More information about the Annotation Marker API can be found in the [Streamline User Guide](https://developer.arm.com/documentation/101816/9-7/Annotate-your-code?lang=en).
 
 {{% notice Note %}}
-You can either build natively on an **Arm platform**, or cross-compile on another architecture using an Arm cross-compiler toolchain.
+You can either build natively on an Arm platform, or cross-compile on another architecture using an Arm cross-compiler toolchain.
 {{% /notice %}}
 
 ### Step 1: Build Streamline Annotation library
 
 Install [Arm DS](https://developer.arm.com/Tools%20and%20Software/Arm%20Development%20Studio) or [Arm Streamline](https://developer.arm.com/Tools%20and%20Software/Streamline%20Performance%20Analyzer) on your development machine first.
 
-Streamline Annotation support code in the installation directory such as *"Arm\Development Studio 2024.1\sw\streamline\gator\annotate"*.
+Streamline Annotation support code is in the installation directory such as `Arm/Development Studio 2024.1/sw/streamline/gator/annotate`.
 
-For installation guidance, refer to the [Streamline installation guide](https://learn.arm.com/install-guides/streamline/).
+For installation guidance, refer to the [Streamline installation guide](/install-guides/streamline/).
 
 Clone the gator repository that matches your Streamline version and build the `Annotation support library`.
 
-The installation step is depends on your development machine.
+The installation step depends on your development machine.
+
+For Arm native build, you can use the following instructions to install the packages.
 
-For Arm native build, you can use following insturction to install the packages.
-For other machine, you need to set up the cross compiler environment by install [aarch64 gcc compiler toolchain](https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads).
-You can refer this [guide](https://learn.arm.com/install-guides/gcc/cross/) for Cross-compiler installation.
+For other machines, you need to set up the cross compiler environment by installing [Arm GNU toolchain](https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads).
+
+You can refer to the [GCC install guide](https://learn.arm.com/install-guides/gcc/cross/) for cross-compiler installation.
 
 {{< tabpane code=true >}}
   {{< tab header="Arm Native Build" language="bash">}}
@@ -40,7 +44,6 @@ You can refer this [guide](https://learn.arm.com/install-guides/gcc/cross/) for
     git clone https://github.com/ARM-software/gator.git
     cd gator
     ./build-linux.sh
-
     cd annotate
     make  
   {{< /tab >}}
@@ -49,20 +52,22 @@ You can refer this [guide](https://learn.arm.com/install-guides/gcc/cross/) for
     apt-get install ninja-build cmake gcc g++ g++-aarch64-linux-gnu curl zip unzip tar pkg-config git
     cd ~
     git clone https://github.com/ARM-software/gator.git
-
     cd gator
     make CROSS_COMPILE=/path/to/aarch64_linux_gcc_tool
   {{< /tab >}}
 {{< /tabpane >}}
 
-Once complete, the static library **libstreamline_annotate.a** will be generated at `~/gator/annotate/libstreamline_annotate.a` and the header file at: `gator/annotate/streamline_annotate.h`
+Once complete, the static library `libstreamline_annotate.a` will be generated at `~/gator/annotate/libstreamline_annotate.a` and the header file is at `gator/annotate/streamline_annotate.h`.
 
 ### Step 2: Integrate Annotation Marker into llama.cpp
 
-Next, we need to install **llama.cpp** to run the LLM model.
-To make the following performance profiling content easier to follow, this Learning Path will use a specific release version of llama.cpp to ensure the steps and results remain consistent.
+Next, you need to install llama.cpp to run the LLM model.
 
-Before the build **llama.cpp**, create a directory `streamline_annotation` and copy the library `libstreamline_annotate.a` and the header file `streamline_annotate.h` into the folder. 
+{{% notice Note %}}
+To make the performance profiling content easier to follow, this Learning Path uses a specific release version of llama.cpp to ensure the steps and results remain consistent.
+{{% /notice %}}
+
+Before building llama.cpp, create a directory `streamline_annotation` and copy the library `libstreamline_annotate.a` and the header file `streamline_annotate.h` into the new directory. 
 
 ```bash
 cd ~
@@ -74,21 +79,21 @@ mkdir streamline_annotation
 cp ~/gator/annotate/libstreamline_annotate.a ~/gator/annotate/streamline_annotate.h streamline_annotation
 ```
 
-To link `libstreamline_annotate.a` library when building llama-cli, adding following lines in the end of `llama.cpp/tools/main/CMakeLists.txt`.
+To link the `libstreamline_annotate.a` library when building llama-cli, add the following lines at the end of `llama.cpp/tools/main/CMakeLists.txt`.
 
 ```makefile
 set(STREAMLINE_LIB_PATH "${CMAKE_SOURCE_DIR}/streamline_annotation/libstreamline_annotate.a")
 target_include_directories(llama-cli PRIVATE "${CMAKE_SOURCE_DIR}/streamline_annotation")
 target_link_libraries(llama-cli PRIVATE "${STREAMLINE_LIB_PATH}")
 ```
 
-To add Annotation Markers to llama-cli, change the llama-cli code **llama.cpp/tools/main/main.cpp** by adding
+To add Annotation Markers to `llama-cli`, change the `llama-cli` code in `llama.cpp/tools/main/main.cpp` by adding the include file:
 
 ```c
 #include "streamline_annotate.h" 
 ```
 
-After the call to common_init(), add the setup macro:
+After the call to `common_init()`, add the setup macro:
 
 ```c
     common_init();
@@ -125,16 +130,16 @@ A string is added to the Annotation Marker to record the position of input token
 
 ### Step 3: Build llama-cli
 
-For convenience, llama-cli is **static linked**.
+For convenience, llama-cli is statically linked.
 
-Firstly, create a new directory `build` understand llama.cpp root directory and go into it.
+Create a new directory `build` under the llama.cpp root directory and change to the new directory: 
 
 ```bash
 cd ~/llama.cpp
-mkdir ./build & cd ./build
+mkdir build && cd build
 ```
 
-Then configure the project by running 
+Next, configure the project.
 
 {{< tabpane code=true >}}
   {{< tab header="Arm Native Build" language="bash">}}
@@ -174,23 +179,20 @@ Then configure the project by running
 {{< /tabpane >}}
 
 
-Set `CMAKE_C_COMPILER` and `DCMAKE_CXX_COMPILER` to your cross compiler path. Make sure that **-march** in `DCMAKE_C_FLAGS` and `CMAKE_CXX_FLAGS` matches your Arm CPU hardware. 
+Set `CMAKE_C_COMPILER` and `CMAKE_CXX_COMPILER` to your cross compiler path. Make sure that -march in `CMAKE_C_FLAGS` and `CMAKE_CXX_FLAGS` matches your Arm CPU hardware. 
 
 
-In this learning path, we run llama-cli on an Arm CPU that supports **NEON Dotprod** and **I8MM** instructions.  
-Therefore, we specify: **armv8.2-a+dotprod+i8mm**.
+With the flags above you can run `llama-cli` on an Arm CPU that supports NEON dot product and 8-bit integer multiply (i8mm) instructions.  
 
-We also specify **-static** and **-g** options:
-- **-static**: produces a statically linked executable, so it can run on different Arm64 Linux/Android environments without needing shared libraries.
-- **-g**: includes debug information, which makes source code and function-level profiling in Streamline much easier.  
+The `-static` and `-g` options are also specified to produce a statically linked executable, so it can run on different Arm64 Linux/Android environments without needing shared libraries and to include debug information, which makes source code and function-level profiling in Streamline much easier.  
 
-so that the llama-cli executable is static linked and with debug info. This makes source code/function level profiling easier and the llama-cli executable runnable on various version of Arm64 Linux/Android.
-
-Now you can build the project by running:
+Now you can build the project using `cmake`: 
 
 ```bash
 cd ~/llama.cpp/build
 cmake --build ./ --config Release
 ```
 
-After the building process, you should find the llama-cli will be generated at **~/llama.cpp/build/bin/** directory.
+After the building process completes, you can find the `llama-cli` in the `~/llama.cpp/build/bin/` directory.
+
+You now have an annotated version of `llama-cli` ready for Streamline.