sized down the model to 70B and the instances to 4xl for ease of use

Author: JoeStech

content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/_index.md

@@ -11,10 +11,10 @@ who_is_this_for: This learning path is for developers with some experience using
 
 learning_objectives: 
     - Set up the main host and worker nodes using llama.cpp
-    - Run a large quantized model (e.g., Llama 3.1 405B) on CPUs in a distributed manner on Arm machines
+    - Run a large quantized model (e.g., Llama 3.1 70B) on CPUs in a distributed manner on Arm machines
 
 prerequisites:
-    - Three AWS c8g.16xlarge instances with at least 2TB EBS space.
+    - Three AWS c8g.4xlarge instances with at least 500GB EBS space.
     - Python installed on the AWS instances.
     - Access to Meta’s gated repository for the Llama 3.1 model family, with a Hugging Face token generated for downloading the models.
     - Familiarity with -> [Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers](/learning-paths/servers-and-cloud-computing/llama-cpu)

content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-1.md

@@ -6,20 +6,20 @@ weight: 2
 layout: learningpathall
 ---
 ## Overview
-This example will run on three AWS Graviton4 c8g.16xlarge instances with 64 cores and 128GB of RAM. The instances should have 2TB disk storage, to store downloaded and quantized model weights.
+This example will run on three AWS Graviton4 c8g.4xlarge instances with 16 cores and 32GB of RAM. The instances should have 500GB disk storage, to store downloaded and quantized model weights.
 
 You will perform these steps in this Learning Path:
 
-1. Download Meta's [405B parameter llama 3.1 model](https://huggingface.co/meta-llama/Llama-3.1-405B).
+1. Download Meta's [70B parameter llama 3.1 model](https://huggingface.co/meta-llama/Llama-3.1-70B).
 2. Download and build llama.cpp, a C++ library that enables efficient inference of LLaMA and similar large language models on CPUs, optimized for local and embedded environments.
 3. Convert Meta's safetensors files to a single gguf file.
 4. Quantize the 16 bit gguf weights file to 4 bit weights.
 5. Load and run the model. 
 
-{{% notice Note %}}The "reading time" mentioned on the Introduction page doesn't include downloading, converting, and requantizing the model. The process mentioned on this page will take 6+ hours. You may skip the model download and quantization if you have a quantized gguf file ready to use.{{% /notice %}}
+{{% notice Note %}}The "reading time" mentioned on the Introduction page doesn't include downloading, converting, and requantizing the model. The process mentioned on this page will take 1-2 hours. You may skip the model download and quantization if you have a quantized gguf file ready to use.{{% /notice %}}
 
 ## Procedure
-First, ensure you have permissions to access to Meta's [405B parameter llama 3.1 model](https://huggingface.co/meta-llama/Llama-3.1-405B).
+First, ensure you have permissions to access to Meta's [70B parameter llama 3.1 model](https://huggingface.co/meta-llama/Llama-3.1-70B).
 
 {{% notice Note %}}
 Remember that you will need to replicate the install steps below on each device. Do NOT replicate the download and quantization step, llama.cpp will send the tensors to the cache.
@@ -29,7 +29,7 @@ Remember that you will need to replicate the install steps below on each device.
 
 ```bash
 apt update
-apt install python3.12-venv
+apt install -y python3.12-venv
 python3 -m venv myenv
 source myenv/bin/activate
 ```
@@ -48,7 +48,6 @@ cmake --build . --config Release
 `llama.cpp` is now built in the `build-rpc/bin` directory.
 Check that `llama.cpp` has built correctly by running the help command:
 ```bash
-cd build-rpc
 bin/llama-cli -h
 ```
 
@@ -58,15 +57,16 @@ Install Huggingface Hub in the virtual environment:
 pip3 install huggingface_hub
 
 ```
-Make a python file and name it download.py:
+Make a python file in a parallel directory to llama.cpp and name it download.py:
 ```bash
+cd ../..
 vi download.py
 ```
 Write the following code to it:
 ```python
 import os
 from huggingface_hub import snapshot_download
-model_id = "meta-llama/Llama-3.1-405B"
+model_id = "meta-llama/Llama-3.1-70B"
 local_dir = "llama-hf"
 # Create the directory if it doesn't exist
 os.makedirs(local_dir, exist_ok=True)
@@ -87,7 +87,7 @@ Following lines installs the files important for conversion to .gguf format.
 pip3 install -r llama.cpp/requirements.txt
 python3 llama.cpp/convert_hf_to_gguf.py llama-hf
 cd llama.cpp/build-rpc
-bin/llama-quantize ../../llama-hf/llama-3.1-405B-F16.gguf Q4_0
+bin/llama-quantize ../../llama-hf/Llama-3.1-70B-F16.gguf Q4_0
 ```
 You may rename the resultant file to model.gguf and use it. There are different quantization options as well, as shown below:
 ```bash

content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-2.md

@@ -6,18 +6,18 @@ weight: 3
 layout: learningpathall
 ---
 ## Cluster overview
-llama.cpp is a C++ library that enables efficient inference of LLaMA and similar large language models on CPUs, optimized for local and embedded environments. Just over a year ago from the publication date of this article, rgerganov’s RPC code was merged into llama.cpp, enabling distributed inference of large LLMs across multiple CPU-based machines—even when the models don’t fit into the memory of a single machine. In this learning path, we’ll explore how to run a 405B parameter model on Arm-based CPUs.
+llama.cpp is a C++ library that enables efficient inference of LLaMA and similar large language models on CPUs, optimized for local and embedded environments. Just over a year ago from the publication date of this article, rgerganov’s RPC code was merged into llama.cpp, enabling distributed inference of large LLMs across multiple CPU-based machines—even when the models don’t fit into the memory of a single machine. In this learning path, we’ll explore how to run a 70B parameter model on Arm-based CPUs.
 
 For the purposes of this demonstration, the following experimental setup will be used:
 - Total number of instances: 3
-- Instance type: c8g.16xlarge
-- Model: model.gguf (Llama-3.1-405B_Q4_0)
+- Instance type: c8g.4xlarge
+- Model: model.gguf (Llama-3.1-70B_Q4_0, ~38GB when quantized to 4 bits)
 
 One of the three nodes will serve as the master node, which physically hosts the model file. The other two nodes will act as worker nodes. In llama.cpp, remote procedure calls (RPC) are used to offload both the model and the computation over TCP connections between nodes. The master node forwards inference requests to the worker nodes, where all the actual computation is performed.
 
 ## Cluster setup
 
-Choose two of the three devices to act as backend workers. If the devices had varying compute capacities, the ones with the highest compute should be selected—especially for a 405B model. However, since all three devices have identical compute capabilities in this case, you can select any two to serve as backend workers.
+Choose two of the three devices to act as backend workers. If the devices had varying compute capacities, the ones with the highest compute should be selected. However, since all three devices have identical compute capabilities in this case, you can select any two to serve as backend workers.
 
 Communication between the master node and the worker nodes occurs through a socket created on each worker. This socket listens for incoming data from the master—such as model parameters, tokens, hidden states, and other inference-related information.
 {{% notice Note %}}The RPC feature in llama.cpp is not secure by default, so you should never expose it to the open internet. To mitigate this risk, ensure that the security groups for all your EC2 instances are properly configured—restricting access to only trusted IPs or internal VPC traffic. This helps prevent unauthorized access to the RPC endpoints.{{% /notice %}}
@@ -36,4 +36,5 @@ Below are the available flag options that can be used with the rpc-server functi
 -m MEM,  --mem MEM        backend memory size (in MB)
 -c,      --cache          enable local file cache
 ```
-Setting the host to 0.0.0.0 might seem counterintuitive given the earlier security warning, but it’s acceptable in this case because the security groups have been properly configured to block any unintended or unauthorized access.
+
+There is currently no way to pre-load tensors onto the workers, the `-c` option will cache tensors during runtime. The first time inference runs the tensors will be distributed to the workers.