add learning path about tuning thread count to improve LLM performance on pytorch-cpu

Author: Your Name

content/learning-paths/servers-and-cloud-computing/tune-pytorch-cpu-perf-with-threads/Background.md

@@ -0,0 +1,132 @@
+---
+title: Background Information
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Background Information
+
+A well-known tradeoff in parallel programming is choosing the right granularity of work. When multiple threads are created to perform a task, the amount of actual computation must justify the cost of coordinating those threads. For example, if a computation is split across many threads, the overhead of creating those threads and synchronizing their results through shared memory can easily outweigh any performance gains if each thread does too little work. The same principle applies to generative AI systems. 
+
+When distribution work across multiple threads, the communication and synchronisation overhead results in more overall work required. This introduces a tradeoff between latency, time to execute a single request, and throughput, number of requests processed in a given time interval. PyTorch looks to automatically find the correct number of threads, however as we will demonstrate, in some cases you may wish to manually fine tune. 
+
+## Multi-threading with PyTorch on CPU
+
+
+The diagram below is taken from the [PyTorch documentation](https://docs.pytorch.org/docs/2.8/notes/cpu_threading_torchscript_inference.html). When running inference, PyTorch uses a thread pool. In PyTorch, we have inter-op parallelism, which is spawning threads to run separate operations in a graph in parallel (e.g., 1 thread for a matmul and another thread for a softmax). Additionally there's intra-op parallelism can be used to spawn multiple threads to work on the same operation. 
+
+![threading-in-pytorch](./pytorch-threading.jpg)
+
+In PyTorch, the `torch.set_num_threads()` [API](https://docs.pytorch.org/docs/stable/generated/torch.set_num_threads.html) is used to set the maximum number of threads to spawn in the Application Thread Pool. As of PyTorch 2.8.0, the default number of threads is equal to the number of CPU cores (see [PyTorch CPU Threading Documentation](https://docs.pytorch.org/docs/2.8/notes/cpu_threading_torchscript_inference.html)). PyTorch looks to find the ideal number of threads as described with the following code snippet taken from the PyTorch source code, [ParallemOpenMP.h](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/ParallelOpenMP.h).
+
+```cpp
+int64_t num_threads = omp_get_num_threads();
+    if (grain_size > 0) {
+      num_threads = std::min(num_threads, divup((end - begin), grain_size));
+    }
+
+...
+inline int64_t divup(int64_t x, int64_t y) {
+  return (x + y - 1) / y;
+}
+```
+
+In PyTorch build with OpenMP, we can set the application thread pool once at runtime using the `OMP_NUM_THREADS` environment variable. The basic demonstration below will show that the default argument on many core systems may not be ideal for your workload. In the use case of running a language model with a low parameter count on a many-core CPU system, this default configuration may not be the most performant configuration. 
+
+## Basic Example
+
+Create a new file called `pytorch_omp_example.py` and paste in the Python script below. We are performing a matrix multiplication in eager mode on two 256-by-256 random matrices. This is clearly a very small operation but we will observe the default behaviour and print the parallel configuration using the `torch.__config__.parallel_info()` function. 
+
+```python
+import os
+import time
+import torch
+
+
+def main():
+    print(f"PyTorch version: {torch.__version__}")
+
+    # Read OMP_NUM_THREADS from the environment
+    omp_threads = os.environ.get("OMP_NUM_THREADS")
+    print(f"OMP_NUM_THREADS in environment: {omp_threads}")
+
+    # If it's set and looks like a number, use it to set PyTorch's intra-op threads
+    if omp_threads and omp_threads.isdigit():
+        torch.set_num_threads(int(omp_threads))
+
+    # Show how many threads PyTorch will actually use for intra-op parallelism
+    print(f"torch.get_num_threads(): {torch.get_num_threads()}\n")
+
+    # A simple operation to illustrate parallelism:
+    size = 256  
+    a = torch.randn(size, size)
+    b = torch.randn(size, size)
+
+    start = time.time()
+    c = a @ b  # matrix multiplication (runs in a parallel region on CPU)
+    end = time.time()
+
+    print(f"Result shape: {c.shape}")
+    print(f"Matrix multiply time: {end - start:.5f} seconds")
+    print(f"\nThreading Information = {torch.__config__.parallel_info()}")
+
+if __name__ == "__main__":
+    main()
+```
+
+
+Running the python script above you will observe the following output. As you can see the number of threads is set to core count of 96 and the time to execute is 2.24 ms. 
+
+```bash
+python pytorch_omp_example.py
+```
+
+```output
+PyTorch version: 2.10.0.dev20251124
+OMP_NUM_THREADS in environment: None
+torch.get_num_threads(): 96
+
+Result shape: torch.Size([256, 256])
+Matrix multiply time: 0.00224 seconds
+
+Threading Information = ATen/Parallel:
+        at::get_num_threads() : 96
+        at::get_num_interop_threads() : 96
+OpenMP 201511 (a.k.a. OpenMP 4.5)
+        omp_get_max_threads() : 96
+Intel(R) MKL-DNN v3.11.0 (Git Hash 0b8a866c009b03f322e6526d7c33cfec84a4a97a)
+std::thread::hardware_concurrency() : 96
+Environment variables:
+        OMP_NUM_THREADS : [not set]
+ATen parallel backend: OpenMP
+```
+
+Now when reduce the number of OpenMP threads using the `OMP_NUM_THREADS` value and observe a reduction is matrix multiply time to 0.64 ms. 
+
+```bash
+OMP_NUM_THREADS=16 python pytorch_omp_example.py
+```
+
+```output
+PyTorch version: 2.10.0.dev20251124
+OMP_NUM_THREADS in environment: 16
+torch.get_num_threads(): 16
+
+Result shape: torch.Size([256, 256])
+Matrix multiply time: 0.00064 seconds
+
+Threading Information = ATen/Parallel:
+        at::get_num_threads() : 16
+        at::get_num_interop_threads() : 96
+OpenMP 201511 (a.k.a. OpenMP 4.5)
+        omp_get_max_threads() : 16
+Intel(R) MKL-DNN v3.11.0 (Git Hash 0b8a866c009b03f322e6526d7c33cfec84a4a97a)
+std::thread::hardware_concurrency() : 96
+Environment variables:
+        OMP_NUM_THREADS : 16
+ATen parallel backend: OpenMP
+```
+
+We will now move on from this trivial example to a LLM. 

content/learning-paths/servers-and-cloud-computing/tune-pytorch-cpu-perf-with-threads/HF-access-token.jpg

@@ -0,0 +1,43 @@
+---
+title: Fine Tune LLM performance on CPU with multithreading
+
+minutes_to_complete: 20
+
+who_is_this_for: ML Engineers looking to fine tune the inference performance of PyTorch running on CPU systems
+
+learning_objectives: 
+    - Understand how PyTorch uses multiple threads for CPU inference and the various tradeoffs involved
+    - Tune the thread count to improve performance for specific models and systems
+
+prerequisites:
+    - Intermediate understanding of Python and PyTorch
+    - Access to an Arm-based system
+
+author: Kieran Hejmadi
+
+### Tags
+skilllevels: Introductory
+subjects: ML
+armips:
+    - Neoverse
+tools_software_languages:
+    - Python
+    - PyTorch
+    - Bash
+operatingsystems:
+    - Linux
+
+
+further_reading:
+    - resource:
+        title: Arm Tool Solutions
+        link: https://github.com/ARM-software/Tool-Solutions/tree/main
+        type: website
+
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1                       # _index.md always has weight of 1 to order correctly
+layout: "learningpathall"       # All files under learning paths have this same wrapper
+learning_path_main_page: "yes"  # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
+---

content/learning-paths/servers-and-cloud-computing/tune-pytorch-cpu-perf-with-threads/Prefill Throughput vs OMP_NUM_THREADS (270M vs 1B).png

@@ -0,0 +1,10 @@
+---
+# ================================================================================
+#       FIXED, DO NOT MODIFY THIS FILE
+# ================================================================================
+weight: 21                  # The weight controls the order of the pages. _index.md always has weight 1.
+title: "Next Steps"         # Always the same, html page title.
+layout: "learningpathall"   # All files under learning paths have this same wrapper for Hugo processing.
+---
+
+If you would like to see other dials available to improve the performance of a LLM on Arm or other types of AI models. Please look at the examples page within the [Tools Solutions repository](https://github.com/ARM-software/Tool-Solutions/tree/main/ML-Frameworks/pytorch-aarch64/examples.)

content/learning-paths/servers-and-cloud-computing/tune-pytorch-cpu-perf-with-threads/_index.md

@@ -0,0 +1,69 @@
+---
+title: Setup Environment
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Build 
+
+In this learning path, we will use Arm’s downstream canary release of PyTorch, which includes ready-to-use examples and scripts. While this release offers access to the latest downstream features, it is intended for experimentation rather than production use.
+
+### 1. Create HuggingFace Account
+
+Create up a [huggingface account](https://huggingface.co/) if you do not already have one. Once created, request access to the [1B](https://huggingface.co/google/gemma-3-1b-it) and [270M](https://huggingface.co/google/gemma-3-270m-it) variants of Google's Gemma-3 model. It will take around 15 minutes to be granted access. 
+
+### 2. Connect to an Arm system and install Docker
+
+lease see our [getting started guide](https://learn.arm.com/learning-paths/servers-and-cloud-computing/csp/) if it is your first time using Arm-based cloud instances. 
+
+In this example I will be using an AWS Graviton 4 (`m8g.24xlarge`)instance running Ubuntu 24.04 LTS. This is based on the Neoverse V2 architecture.
+
+
+Install docker through the [official documentation](https://docs.docker.com/engine/install/ubuntu/) or our [Arm install guide](https://learn.arm.com/install-guides/docker/docker-desktop-arm-linux/). Make sure to follow the post-installation steps. 
+
+
+### 3. Build the PyTorch-AArch64 Docker Container
+
+Connect to the Arm instance and run the following to clone the repository.
+
+```bash
+git clone https://github.com/ARM-software/Tool-Solutions.git
+cd Tool-Solutions/ML-Frameworks/pytorch-aarch64/
+```
+Run the following bash script to build the container
+
+```bash
+./build.sh -n $(nproc - 1)
+```
+
+> **Note**: On a 96-core instance `m8g.24` AWS this will take approximately 20 minutes to build. 
+
+Once the build has finished, run the following command replacing `<version>` with the version of torch and torchao just built. 
+
+```bash
+./dockerize.sh ./results/torch-<version>linux_aarch64.whl ./results/torchao-<version>-py3-none-any.whl 
+```
+
+You should see the following output in your terminal, confirming that you are in the correct directory inside the Docker container.
+
+```outpu
+aarch64_pytorch ~> 
+```
+
+### 5. Login to HuggingFace
+
+
+
+Create a new `read` token on HuggingFace by clicking [this link](https://huggingface.co/settings/tokens/new?tokenType=read). 
+
+![hf-token](./HF-access-token.jpg)
+
+Provide a suitable token name, press create token and copy the generated token value. From within docker container, enter the following command and paste the token to login.
+
+```bash
+huggingface-cli login
+```
+
+> **Note**: The login will not persist once the docker session has ended