Tech review of GPT-2 and Performix

Author: jasonrandrews

content/learning-paths/servers-and-cloud-computing/performix-instruction-mix/_index.md

@@ -1,6 +1,10 @@
 ---
 title: Profile GPT-2 instruction mix with Arm Performix
 
+draft: true
+cascade:
+    draft: true
+
 description: Learn how to profile GPT-2 inference on Arm Neoverse with the Arm Performix Instruction Mix recipe, identify scalar versus vector execution patterns, and improve throughput with NEON, SVE, and KleidiAI kernels.
 
 minutes_to_complete: 45
@@ -17,7 +21,7 @@ prerequisites:
     - Access to Arm Performix configured with a remote Arm Linux target. For setup, see the [Arm Performix install guide](/install-guides/performix/)
     - Basic understanding of C++ and compiler optimization
     - Basic understanding of matrix multiplication
-    - Basic understanding of writing SIMD code with Neon and/or SVE.
+    - Basic understanding of writing SIMD code with Neon or SVE
 
 author:
     - Kieran Hejmadi
@@ -30,7 +34,7 @@ armips:
     - Neoverse
 tools_software_languages:
     - Arm Performix
-    - C++
+    - C
     - LLM
     - NEON
     - SVE

content/learning-paths/servers-and-cloud-computing/performix-instruction-mix/how-to-1.md

@@ -1,12 +1,12 @@
 ---
-title: Background
+title: Understand instruction mix profiling
 weight: 2
 
 ### FIXED, DO NOT MODIFY
 layout: learningpathall
 ---
 
-## What the instruction mix recipe shows
+## Instruction mix overview
 
 The Arm Performix Instruction Mix recipe shows the types and proportions of machine instructions your workload executes at runtime and in static analysis, so you can see how efficiently your code uses Arm CPU hardware resources.
 
@@ -25,20 +25,20 @@ The instruction mix result gives you two complementary views:
 
 Together, these views help you verify whether architecture-specific features are actually active in hot code paths.
 
-## Why instruction mix is useful
+## Detecting missed vectorization
 
 Instruction mix is useful when you need to confirm that performance-critical code uses Arm CPU features effectively. This is especially helpful when you are, for example, validating the effectiveness of compiler autovectorization.
 
 For example, if a hot function is mostly scalar at runtime when you expected NEON or SVE activity, that often indicates missed vectorization opportunities. You can then focus optimization work on compiler flags, data layout, loop structure, and kernel implementation to improve throughput where it matters most.
 
-## Why use a GPT-2 workload
+## GPT-2 as a test workload
 
-In this Learning Path, you run the [GPT-2 Medium](https://huggingface.co/openai-community/gpt2-medium) model on a minimal C++ inference engine to analyze instruction mix and throughput. This model is available under a [modified MIT License](https://github.com/openai/gpt-2/blob/master/LICENSE). You will confirm that matrix multiplication (`matmul`) is the hot path, then compare how scalar, NEON, and SVE implementations change instruction behavior and token generation speed. 
+You can run the [GPT-2 Medium](https://huggingface.co/openai-community/gpt2-medium) model on a minimal C++ inference engine to analyze instruction mix and throughput. This model is available under a [modified MIT License](https://github.com/openai/gpt-2/blob/master/LICENSE). You will confirm that matrix multiplication (`matmul`) is the hot path, then compare how scalar, NEON, and SVE implementations change instruction behavior and token generation speed. 
 
 This example implements only the forward inference path, with no back propagation or training. You do not need to understand the full transformer architecture to complete this Learning Path. Familiarity with matrix multiplication is enough. For background on GPT-2, see the original 2019 paper, [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
 
 You will also try implementing your own `matmul` kernels that target NEON and SVE, then use instruction mix data to verify that these vector paths are active and improving throughput.
 
-## What you've learned and what's next
+## Next steps
 
-In this section, you learned what instruction mix represents and why it is useful for LLM inference optimization on Arm. Next, you will set up the GPT-2 example, build the binaries, and run a baseline test.
+You now know what instruction mix represents and why it matters for LLM inference optimization on Arm. Next, you'll set up the GPT-2 example, build the binaries, and run a baseline test.

content/learning-paths/servers-and-cloud-computing/performix-instruction-mix/how-to-2.md

@@ -24,7 +24,7 @@ git checkout tags/v0.0.2
 
 The C++ runtime expects exported model binaries. Create a Python virtual environment, install dependencies, and export GPT-2 Medium weights and vocabulary:
 
-This Learning Path uses [openai-community/gpt2-medium on Hugging Face](https://huggingface.co/openai-community/gpt2-medium), which corresponds to the GPT-2 Medium model from the original OpenAI GPT-2 release in 2019. The model has 355 million parameters, and in this workflow it runs with unquantized FP32 (32-bit floating-point) weights.
+The workload uses [openai-community/gpt2-medium on Hugging Face](https://huggingface.co/openai-community/gpt2-medium), which corresponds to the GPT-2 Medium model from the original OpenAI GPT-2 release in 2019. The model has 355 million parameters, and in this workflow it runs with unquantized FP32 (32-bit floating-point) weights.
 
 ```bash
 python3 -m venv venv
@@ -105,8 +105,10 @@ Run the scalar baseline binary:
 ./build/gpt2 --model gpt2-medium "Once upon a time" -n 20
 ```
 
+The output is:
+
 ![Animated terminal output showing GPT-2 baseline inference running on Arm Linux, including generated text and the final tokens-per-second summary.#center](./gpt2-baseline.gif "GPT-2 baseline runtime output on Arm Linux")
 
-## What you've learned and what's next
+## Next steps
 
-You now have a working baseline binary and model files. Next, you will use the Instruction Mix recipe in Arm Performix to inspect static disassembly and dynamic runtime behavior.
+You now have a working baseline binary and model files. Next, you'll use the Instruction Mix recipe in Arm Performix to inspect static disassembly and dynamic runtime behavior.

content/learning-paths/servers-and-cloud-computing/performix-instruction-mix/how-to-3.md

@@ -12,10 +12,12 @@ Before you optimize, identify where the application spends most of its time. Use
 
 Open Arm Performix and select the **Code Hotspots** recipe. If this is your first run on the target, complete tool deployment as prompted.
 
-Set the launch command to your baseline binary with the number of tokens (`-n`) set to 150 as per the command below. This value keeps startup overhead small compared to inference time, so the profile minimizes the time taken to load the model weights:
+Set the launch command to your baseline binary with the number of tokens (`-n`) set to 150 as per the command below. This value keeps startup overhead small compared to inference time, so the profile minimizes the time taken to load the model weights.
 
-```out
-<path to repository>/build/gpt2 --model gpt2-medium "Once upon a time" -n  150
+Copy and paste the command below as the Performix Workload. If your path is different adjust it to point to `gpt2`. 
+
+```console
+/home/ubuntu/GPT-2-Example/build/gpt2 --model gpt2-medium "Once upon a time" -n  150
 ```
 
 ![Arm Performix Code Hotspots recipe configuration showing launch arguments for the GPT-2 baseline run with -n 150 to emphasize inference runtime.#center](./code_hotspot.webp "Code Hotspots recipe configuration for GPT-2 baseline")
@@ -38,13 +40,16 @@ The Instruction Mix recipe helps fill this gap.
 
 ## Configure the Instruction Mix recipe
 
-Open Arm Performix and select the **Instruction Mix** recipe. If this is your first run on the target, complete tool deployment as prompted.
+Open Arm Performix and select the **Instruction Mix** recipe. 
+
 Set the launch command to your baseline binary with the same runtime arguments used for baseline testing:
 
-```output
-</path/to/GPT-2-Example>/build/gpt2 --model gpt2-medium "Once upon a time" -n 150`
+```console
+/home/ubuntu/GPT-2-Example/build/gpt2 --model gpt2-medium "Once upon a time" -n  150
 ```
 
+Note about number of counters.
+
 Use the same model and prompt arguments as your baseline terminal run so the measurements are comparable.
 
 ![Arm Performix recipe setup screen showing Instruction Mix recipe selected with launch settings configured for the GPT-2 baseline executable.#center](./configuring-performix.webp "Configure Arm Performix Instruction Mix recipe")
@@ -63,6 +68,6 @@ Then inspect dynamic analysis bar chart to see where sampled runtime work is con
 
 Finally, in dynamic functions, you can break down operation types to individual functions. This is particularly useful when no single function dominates the profile, allowing you to inspect dynamic instruction patterns for specific functions.
 
-## What you've learned and what's next
+## Next steps
 
-You used Instruction Mix to confirm that baseline runtime is dominated by scalar-heavy `matmul` execution. Next, you will compare updated instruction mix and throughput across scalar, NEON, SVE, and KleidiAI variants.
+You used Instruction Mix to confirm that baseline runtime is dominated by scalar-heavy `matmul` execution. Next, you'll compare updated instruction mix and throughput across scalar, NEON, SVE, and KleidiAI variants.

content/learning-paths/servers-and-cloud-computing/performix-instruction-mix/how-to-4.md

@@ -1,5 +1,5 @@
 ---
-title: Optimize
+title: Optimize matmul with vector intrinsics
 weight: 5
 
 ### FIXED, DO NOT MODIFY
@@ -66,6 +66,6 @@ Restart your coding assistant, then prompt it to run Performix Instruction Mix a
 
 ![Screenshot of a coding assistant prompt configured to use Arm MCP Server tools for running Performix recipes and analyzing matmul_user optimization opportunities in the GPT-2 workload.#center](./mcp-performix-prompt.webp "Coding assistant prompt for Performix analysis through Arm MCP Server")
 
-## What you've learned and what's next
+## Next steps
 
-In this optional section, you implemented and profiled a custom `matmul_user` kernel using the same workflow you used for baseline analysis. Next, you will compare instruction mix and throughput across scalar, NEON, SVE, and KleidiAI variants.
+In this optional section, you implemented and profiled a custom `matmul_user` kernel using the same workflow you used for baseline analysis. Next, you'll compare instruction mix and throughput across scalar, NEON, SVE, and KleidiAI variants.

content/learning-paths/servers-and-cloud-computing/performix-instruction-mix/how-to-5.md

@@ -6,7 +6,7 @@ weight: 7
 layout: learningpathall
 ---
 
-## Confirm instruction-mix changes after vectorization
+## Verify vectorization with instruction mix
 
 In the earlier build step, you created `gpt2_neon` and `gpt2_sve`. These binaries use the reference solutions in `matmul_neon.cpp` and `matmul_sve.cpp`, respectively. Run the `gpt2_neon` binary with the following command to observe the speedup. 
 
@@ -56,7 +56,7 @@ For a full-page view, open [Godbolt session with all three matmul kernels](https
 
 Run the provided comparison script to measure tokens per second across all available binaries:
 
-```bash bash { command_line="user@host | 2-30"}
+```bash bash { command_line="ubuntu@ip | 2-30"}
 ./compare_gpt2_variants.sh
 Model: gpt2-medium
 Prompt: Once upon a time
@@ -80,4 +80,4 @@ run 1: 3.04859 tok/s
 avg: 3.048590 tok/s
 ```
 
-These results show that intrinsics increase throughput from about 3 tok/s in the scalar baseline to about 13.9 tok/s with SVE. Next, you will use optimized libraries to push performance further.
+These results show that intrinsics increase throughput from about 3 tok/s in the scalar baseline to about 13.9 tok/s with SVE. Next, you will use optimized libraries to push performance further.

content/learning-paths/servers-and-cloud-computing/performix-instruction-mix/how-to-6.md

@@ -6,7 +6,7 @@ weight: 8
 layout: learningpathall
 ---
 
-## Understand why KleidiAI improves matmul
+## KleidiAI packing and tiling
 
 Getting close to peak matmul performance is complex, especially in transformer inference where one kernel runs many times per token. Raw SIMD intrinsics can accelerate execution, but higher-level algorithms can improve matrix multiplication through techniques such as packing and tiling.
 
@@ -15,7 +15,7 @@ Getting close to peak matmul performance is complex, especially in transformer i
 - *Tiling* breaks large matrices into smaller blocks that fit better in cache, so data is reused more often and memory bandwidth pressure is lower.
 Arm created [KleidiAI](https://github.com/ARM-software/kleidiai) to provide the fastest Arm CPU microkernels on packing and tiled matrix multiplication so you can use these optimizations without writing and tuning every low-level kernel yourself.
 
-## Trace the KleidiAI path in this GPT-2 example
+## KleidiAI integration in the GPT-2 example
 
 The file `src/kernels/matmul_kai_sve.cpp` is the runtime bridge between your model code and a KleidiAI float32 SVE microkernel. Since packing is a preprocessing step, this workflow uses a slightly modified inference engine, `gpt2_kai_sve.cpp`. It builds a `kai_matmul_clamp_f32_f32_f32p_ukernel` function table and binds function pointers such as:
 
@@ -27,7 +27,7 @@ The `kai_run_matmul_clamp_f32_f32_f32p4vlx1b_6x4vl_sve_mla` is our entry. As per
 
 For more details on KleidiAI, please refer to the [official GitLab repository](https://gitlab.arm.com/kleidi/kleidiai).
 
-## Read the source code: key steps
+## Key source code
 
 All matmul implementations in `src/kernels/` follow the same high-level pattern from `matmul.h`: they compute float32 output from float32 input vectors and weights. The KleidiAI variant keeps this behavior but changes the RHS argument to a packed buffer (`const uint8_t* rhs_packed`) so the compute path can consume prepacked tiles.
 
@@ -62,11 +62,11 @@ The execution flow is:
 In `src/gpt2_kai_sve.cpp`, runtime code prepares packed weights once with `kai_run_rhs_pack_kxn_x32p4vlx1b_x32_x32_sve`, stores them in `PackedWeights`, and calls `kernels::matmul_kai_sve(...)` at runtime. This is why the runtime file and `matmul_kai_sve.cpp` must match: the runtime produces packed buffers in the format expected by the same f32 ukernel family.
 
 
-## Compare SVE intrinsics and KleidiAI
+## SVE intrinsics versus KleidiAI
 
 Run the comparison script from the repository root with the following command.
 
-```bash { command_line="user@host | 2-30"}
+```bash { command_line="ubuntu@ip | 2-30"}
 ./compare_gpt2_variants.sh kai
 Model: gpt2-medium
 Prompt: Once upon a time
@@ -103,7 +103,7 @@ avg: 3.047840 tok/s
 You can increase throughput by running the KleidiAI path with multiple matmul threads. For this 355M model, tune `--matmul-threads` heuristically on your target system to find the optimal value. For our Graviton 3 instance, we observe a max token generation speed of 34.5 token/s with 4 threads.
 
 
-``` bash { command_line="user@host | 3-50"}
+``` bash { command_line="ubuntu@ip | 3-50"}
 cd build
 ./gpt2_kai_sve --model gpt2-medium "Once upon a time" -n 150 --matmul-threads 4
 Weights path: /home/ubuntu/GPT-2-DEMO/GPT-2-Example/models/gpt2-medium/weights.bin
@@ -139,4 +139,6 @@ They would be kicked out of the village, and they would suffer and die as punish
 
 ## Summary
 
-In this Learning Path, you used Arm Performix Instruction Mix to detect scalar-heavy hot paths, validated vectorization changes with static and dynamic evidence, and compared baseline, Neon, SVE, and KleidiAI-backed matmul implementations. This workflow is transferable to your own codebase: use instruction mix to detect missed vectorization and other unexpected instruction-balance patterns, validate changes with static and runtime evidence, and then tune to meet your performance requirements.
+You used Arm Performix Instruction Mix to detect scalar-heavy hot paths, validated vectorization changes with static and dynamic evidence, and compared baseline, Neon, SVE, and KleidiAI-backed matmul implementations. Starting from roughly 3 tok/s with scalar code, you reached about 15 tok/s with SVE intrinsics and 35 tok/s with KleidiAI packing, tiling, and multithreading.
+
+This workflow is transferable to your own codebase: use instruction mix to detect missed vectorization and other unexpected instruction-balance patterns, validate changes with static and runtime evidence, and then tune to meet your performance requirements.