Skip to content

Commit 1c4af8b

Browse files
Updated index
1 parent 7340247 commit 1c4af8b

1 file changed

Lines changed: 8 additions & 12 deletions

File tree

  • content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp

content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/_index.md

Lines changed: 8 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,19 @@
11
---
22
title: Distributed inference using llama.cpp
33

4-
draft: true
5-
cascade:
6-
draft: true
7-
84
minutes_to_complete: 30
95

10-
who_is_this_for: This learning path is for developers with some experience using llama.cpp who want to learn about distributed inference.
6+
who_is_this_for: This Learning Path is for developers with some experience using llama.cpp who want to learn distributed inference.
117

128
learning_objectives:
13-
- Set up the main host and worker nodes using llama.cpp
14-
- Run a large quantized model (e.g., Llama 3.1 405B) on CPUs in a distributed manner on Arm machines
9+
- Set up a main host and worker nodes with llama.cpp
10+
- Run a large quantized model (for example, Llama 3.1 405B) with distributed CPU inference on Arm machines
1511

1612
prerequisites:
17-
- Three AWS c8g.16xlarge instances with at least 2TB EBS space.
18-
- Python installed on the AWS instances.
19-
- Access to Metas gated repository for the Llama 3.1 model family, with a Hugging Face token generated for downloading the models.
20-
- Familiarity with -> [Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers](/learning-paths/servers-and-cloud-computing/llama-cpu)
13+
- Three AWS c8g.16xlarge instances with at least 2 TB of EBS storage
14+
- Python 3 installed on each instance
15+
- Access to Meta's gated repository for the Llama 3.1 model family and a Hugging Face token to download models
16+
- Familiarity with [Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers](/learning-paths/servers-and-cloud-computing/llama-cpu)
2117
- Familiarity with AWS
2218

2319
author: Aryan Bhusari
@@ -38,7 +34,7 @@ operatingsystems:
3834

3935
further_reading:
4036
- resource:
41-
title: Llama.cpp rpc-server code
37+
title: llama.cpp RPC server code
4238
link: https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc
4339
type: Code
4440

0 commit comments

Comments
 (0)