Skip to content

Commit b441a78

Browse files
committed
add a note about how long it takes to load tensors on worker nodes
1 parent f73e5f2 commit b441a78

1 file changed

Lines changed: 5 additions & 0 deletions

File tree

  • content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp

content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-3.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,11 @@ Finally, you can execute the following command, to execute distributed inference
2727
```bash
2828
bin/llama-cli -m ../../model.gguf -p "Tell me a joke" -n 128 --rpc "$worker_ips" -ngl 999
2929
```
30+
31+
{{% notice Note %}}
32+
It will take a significant amount of time (~30 minutes) to load the tensors on the worker nodes. Pre-loaded tensors are a current development request for llama.cpp.
33+
{{% /notice %}}
34+
3035
Here are short definitions of the flags used in above command:
3136
-n => Number of maximum output tokens
3237
--rpc => list of backend workers

0 commit comments

Comments
 (0)