You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/_index.md
+8-12Lines changed: 8 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,23 +1,19 @@
1
1
---
2
2
title: Distributed inference using llama.cpp
3
3
4
-
draft: true
5
-
cascade:
6
-
draft: true
7
-
8
4
minutes_to_complete: 30
9
5
10
-
who_is_this_for: This learning path is for developers with some experience using llama.cpp who want to learn about distributed inference.
6
+
who_is_this_for: This Learning Path is for developers with some experience using llama.cpp who want to learn distributed inference.
11
7
12
8
learning_objectives:
13
-
- Set up the main host and worker nodes using llama.cpp
14
-
- Run a large quantized model (e.g., Llama 3.1 405B) on CPUs in a distributed manner on Arm machines
9
+
- Set up a main host and worker nodes with llama.cpp
10
+
- Run a large quantized model (for example, Llama 3.1 405B) with distributed CPU inference on Arm machines
15
11
16
12
prerequisites:
17
-
- Three AWS c8g.16xlarge instances with at least 2TB EBS space.
18
-
- Python installed on the AWS instances.
19
-
- Access to Meta’s gated repository for the Llama 3.1 model family, with a Hugging Face token generated for downloading the models.
20
-
- Familiarity with -> [Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers](/learning-paths/servers-and-cloud-computing/llama-cpu)
13
+
- Three AWS c8g.16xlarge instances with at least 2 TB of EBS storage
14
+
- Python 3 installed on each instance
15
+
- Access to Meta's gated repository for the Llama 3.1 model family and a Hugging Face token to download models
16
+
- Familiarity with [Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers](/learning-paths/servers-and-cloud-computing/llama-cpu)
0 commit comments